Why crawler reliability lives outside the scraper

Caneren Altungül — Fri, 04 Sep 2026 14:08:52 +0000

When people think about building a crawler, most of the attention goes to the part that fetches a page and extracts data.

In practice, that is often the easy part.

The harder problems start around the crawler.

What happens when a request fails? What happens when the process crashes halfway through a job? How do you avoid processing the same record twice? Who owns retries? How do workers coordinate? What happens to a message that keeps failing? How do you shut down cleanly? How do you distinguish cancellation from failure? How do you recover work after a lease expires?

These problems are not really scraping problems.

They are reliability and execution problems.

That distinction is what eventually led me to build Enterprise Crawler Framework (ECF).

I originally needed the same reliability infrastructure repeatedly while working on a larger data-ingestion project. Instead of solving lifecycle, retry, state, worker coordination, and failure handling again for every crawler, I started separating those concerns from the source-specific crawling logic.

The result became ECF.

The crawler should contain crawler logic

My preferred boundary is simple:

You write the source-specific logic. The framework handles the reusable infrastructure around it.

A minimal bot can look like this:

from enterprise_crawler import BaseBot, Crawler


class HelloBot(BaseBot):
    def execute(self) -> None:
        print("Hello from Enterprise Crawler Framework!")
        self.mark_record_processed()


with HelloBot(bot_name="hello-bot") as bot:
    crawler = Crawler(bot)
    result = crawler.run()

print(f"status={result.status.value}")
print(f"records_processed={result.records_processed}")

The framework does not know what website, API, feed, document source, or business domain the bot belongs to.

That belongs to the application.

The framework is responsible for the execution infrastructure around that logic.

This separation sounds simple, but it becomes more valuable as a crawler grows beyond a one-off script.

Reliability needs explicit ownership

One of the easiest ways to make a crawler unreliable is to let responsibilities blur together.

Retries are a good example.

An HTTP retry answers a transport-level question:

Should this HTTP request be attempted again?

A worker retry answers a different question:

Should this unit of work be scheduled for another execution attempt?

Those are not the same decision.

They may have different retry limits, delays, failure policies, and operational consequences.

A temporary network error might justify repeating a request immediately.

A failed unit of work might need to be retried several minutes later.

A permanently invalid record might not deserve a retry at all.

ECF therefore keeps HTTP retry concerns separate from event and worker retry policy.

The goal is not to make retries more complicated.

The goal is to make ownership explicit.

Lifecycle matters more than it first appears

A crawler that works once is easy.

A crawler that can initialize resources, execute work, handle cancellation, finalize correctly, clean up, and shut down predictably is harder.

ECF treats crawler execution as a lifecycle rather than a single function call.

The application puts its source-specific behavior inside execute() while the framework controls the surrounding runtime lifecycle.

That also means cancellation can have its own semantics.

A cancelled run is not automatically a failed run.

That distinction matters when a crawler is intentionally stopped, a deployment is being replaced, or a worker is being drained.

Failure and cancellation may both stop execution, but they describe different operational situations.

Making that difference visible produces better runtime behavior and better diagnostics.

Optional infrastructure should stay optional

Not every crawler needs a database.

Not every crawler needs plugins.

Not every crawler needs workers or a durable event queue.

A small one-shot bot should not create persistent state just because the framework supports persistence.

This is an important design principle in ECF:

Simple use cases should remain simple, while stronger infrastructure should be available when the workload actually needs it.

Storage is therefore optional.

Plugin management is optional.

A simple bot can run without automatic SQLite or plugin side effects.

When persistence is required, ECF provides storage and durable event infrastructure that can be composed into the application.

The framework should not force production complexity onto applications that do not need it.

Queues are mostly about failure semantics

Putting a message into a queue is not the difficult part.

The difficult questions come afterward.

Who owns the message?

How is that ownership represented?

What happens if the worker disappears?

When does the work become available again?

What happens when the handler fails?

When should another retry occur?

What happens when retry attempts are exhausted?

These are the behaviors that determine whether a queue is actually reliable.

The current ECF event subsystem supports both in-memory and durable SQLite-backed queues.

It includes claims, claim tokens, leases, lease recovery, scheduled retries, retry policies, exponential backoff, optional jitter, workers, and dead-letter handling.

One distinction is especially important:

A lease is not a retry delay.

A lease protects temporary ownership while a worker is processing a message.

A retry delay determines when failed work becomes eligible for another attempt.

Combining those concepts can create subtle recovery bugs.

Keeping them separate makes worker behavior easier to reason about.

Claim tokens protect ownership

A message identifier alone is not enough to prove that a worker still owns a message.

Imagine this sequence:

A worker claims a message.

Its lease expires.

Another worker recovers and claims the same message.

The original worker later attempts to acknowledge it.

If acknowledgement requires only the message ID, the stale worker could incorrectly finalize work that it no longer owns.

That is why ECF queue operations use both a message identifier and a claim token for ownership-sensitive operations.

The token represents the current claim, not merely the identity of the message.

This is a small detail, but these small details are where durable worker systems usually become difficult.

Retry delay should survive process restarts

A retry mechanism is not truly durable if its timing exists only in memory.

If a worker decides that a message should retry in thirty seconds and the process crashes five seconds later, the retry schedule should not disappear.

For durable queues, ECF persists retry eligibility.

A scheduled retry therefore remains scheduled even when the process restarts.

This also means a future-due message should not block other work that is already eligible.

The queue needs to reason about both ordering and eligibility.

Again, this is not scraping logic.

It is infrastructure around scraping logic.

Dead-letter handling should fail closed

Eventually, some messages should stop retrying.

Maybe the retry budget has been exhausted. Maybe the failure policy says the work should be discarded from normal processing. Maybe an operator needs to inspect it manually.

That is where dead-letter handling becomes important.

But there is another failure boundary hiding inside that operation.

Suppose the system removes a failed message from the source queue first and then attempts to store it in the dead-letter queue.

If the dead-letter write fails, the work is gone.

The source no longer owns it and the dead-letter store never received it.

That is silent data loss.

ECF uses the safer ordering:

Dead-letter storage must succeed before the source message is finalized.

If dead-letter storage fails, the original work remains unresolved instead of silently disappearing.

For ingestion systems, temporary unresolved work is usually preferable to irreversible loss.

"Exactly once" is usually the wrong promise

ECF deliberately does not claim exactly-once processing.

Durable systems can crash at awkward boundaries.

A process might fail before execution.

It might fail after execution but before acknowledgement.

It might perform an external side effect and crash before recording that the operation succeeded.

A lease might expire while a slow operation is still running.

A framework can provide strong ownership semantics and reduce unnecessary duplicate execution, but applications still need to think about idempotency when external side effects are involved.

I would rather expose that reality than make an exactly-once promise that the framework cannot honestly guarantee.

In practical systems, explicit failure semantics are more useful than stronger-sounding guarantees.

Plugins should not execute during discovery

Plugin systems create another trust and lifecycle boundary.

There is an important difference between asking:

What plugins are available?

and asking:

Load and execute this plugin.

Discovery should not automatically run third-party code.

ECF therefore separates plugin discovery from plugin loading.

Discovery works from entry-point metadata and does not import the plugin.

Loading happens later.

Lifecycle management and registration happen after that.

Conceptually:

DiscoveredPlugin
        ↓
LoadedPlugin
        ↓
RegisteredPlugin

These are deliberately different states.

That makes it easier to reason about when third-party code actually enters the process.

Processing should stay domain-independent

Crawler frameworks can easily become collections of application-specific behavior.

I wanted to avoid that.

ECF contains generic processing primitives and pipelines, but the framework itself does not know anything about a particular business domain.

It should not contain rules for a specific website, government source, e-commerce platform, legal dataset, or customer application.

Those belong outside the framework.

The same rule applies to provider-specific integrations.

Generic infrastructure can belong in ECF.

Application knowledge should remain in the application.

The public API should remain small

Another design choice is keeping the top-level API intentionally small.

The current public top-level API exposes:

from enterprise_crawler import (
    BaseBot,
    Crawler,
    ExecutionResult,
    ExecutionStatus,
)

along with framework and version metadata.

Not every internal class needs to become a permanent public contract.

Once an API becomes public, changing it becomes a compatibility problem.

Keeping the public surface small gives the internals room to evolve without forcing unnecessary breaking changes on users.

The CLI follows the same principle

ECF currently provides a deliberately small CLI.

For example:

enterprise-crawler --version
enterprise-crawler version
enterprise-crawler doctor
enterprise-crawler plugins list
enterprise-crawler plugins inspect <PLUGIN_NAME>

There is currently no run, new, init, or project-scaffolding command.

Those may sound useful, but I do not want to add commands simply because frameworks are expected to have them.

The CLI should grow because real workflows need it, not because a longer command list looks more complete.

The framework should stay smaller than the applications built with it

This is probably the most important product constraint I am trying to preserve.

ECF is not intended to become every part of a scraping stack.

It is not a proxy provider.

It is not a CAPTCHA-solving service.

It is not a browser cloud.

It is not a hosted crawler platform.

It is not an application-specific crawler.

It also does not currently provide distributed queue backends such as Redis, Kafka, RabbitMQ, or SQS.

Those systems may eventually become relevant.

But technical possibility is not the same thing as product requirement.

Adding infrastructure before real users need it would increase maintenance cost and complexity without proving that the framework becomes more useful.

The model today is intentionally straightforward:

Developer
    ↓
Enterprise Crawler Framework
    ↓
Custom Bot
    ↓
External Data Source

The developer owns the crawler and the execution environment.

ECF provides reusable infrastructure around it.

Where the project is now

Enterprise Crawler Framework is currently at v1.0.1.

It is open source under the MIT License and supports Python 3.11 and newer.

You can install it directly from PyPI:

python -m pip install enterprise-crawler-framework

If you want to pin the current release:

python -m pip install enterprise-crawler-framework==1.0.1

The current Community release includes runtime lifecycle infrastructure, HTTP and session handling, optional storage, generic processing primitives, plugin discovery/loading/management, an in-memory event queue, a durable SQLite event queue, workers, claims and leases, scheduled retries, exponential backoff with optional jitter, and dead-letter handling.

The source code is available on GitHub:

https://github.com/canerenaltungul/enterprise-crawler-framework

The package is available on PyPI:

https://pypi.org/project/enterprise-crawler-framework/

What I do not know yet

Publishing the framework is not the same thing as knowing what should come next.

There are many technically interesting directions I could take it.

Distributed workers.

More queue backends.

Observability.

Deployment tooling.

More integrations.

A larger CLI.

Hosted services.

But building those now would mostly mean guessing.

I would rather see what happens when people actually try to use the current framework.

That is the stage the project is in now.

What I want to learn from real users

If you build crawlers or data-ingestion systems, I would be especially interested in three things:

Can you understand the framework boundary from the README and quickstart?
Can you get a small bot running without fighting the framework?
What reliability or operational problem would stop you from using something like this for a real crawler?

I am also interested in negative feedback.

If something feels unnecessary, over-engineered, confusing, or simply not useful, that is valuable information.

The goal is not to turn every request into a feature.

The goal is to find the problems that repeat across real users.

If several people independently run into the same limitation, that is a much stronger roadmap signal than another feature I can invent on my own.

For now, I want to keep the framework focused on one idea:

You write the crawler. ECF provides the infrastructure around it.

DEV Community: Caneren Altungül