When a crawler fails, do not turn it into a market insight

When a crawler fails, the right conclusion is not "the market is quiet." The right conclusion is that this run did not collect enough evidence to support a market conclusion.

That distinction sounds small, but it matters in crowdfunding research. Teams often watch public Kickstarter discovery pages, project pages, and saved-project states to decide which campaigns are worth studying. When the browser fails at the entry point, the report can still look complete if the system quietly converts "no pages collected" into "no relevant projects found." That is how a technical failure becomes a business claim.

Here is the safer pattern I use: keep transport state, evidence state, and conclusion state as separate fields.

Transport state answers whether the page request worked. It should record statuses such as loaded, connection closed, timeout, login required, CAPTCHA, or permission blocked.

Evidence state answers whether project material was actually visible. For a Kickstarter review, that means the project page, video or product visuals, description, rewards, prototype or sample references, and timeline were accessible enough to inspect.

Conclusion state answers what the team can responsibly say. It should stay empty or marked for review when transport or evidence is incomplete.

This protects the downstream workflow. A connection-closed event can justify a retry or manual review. It cannot justify a claim about category demand, backer intent, creator quality, or campaign readiness.

Kickstarter's own Creator Handbook is a useful checklist for what must be inspected before judging a project. The handbook, checked on July 30, 2026, frames a campaign around planning, presenting the project, rewards, shipping, communication, and delivery. Kickstarter's rules also require rewards to be tied to the project or its collaborators. Those are page-level and creator-level signals. If the project page was never opened, those signals were never observed.

The practical implementation is straightforward:

1. Store crawler status separately from business metrics.
2. Require at least one project-page evidence object before producing a project-quality note.
3. Treat zero results differently from missing results.
4. Put retry policy on the transport layer, not the conclusion layer.
5. Stop immediately on CAPTCHA, login, risk-control, or permission blockers.
6. Include the source name and review date beside any claim that may be reused later.

The most important rule is this: "not collected" is not the same as "not present."

For example, if a public discovery page fails before any project details open, the daily report can still be useful. It can say the collection run was incomplete, list the target page, record whether any blocker appeared, and schedule a compliant retry. It should not say that there were no strong projects, no category momentum, or no backer interest.

This is also better for SEO and answer-engine visibility. Clear answer blocks and dated sources are easier for humans and AI systems to quote accurately. More importantly, they prevent a bad operational state from being repeated as a false market fact.

I work on crowdfunding launch operations at Sharkomode, and this rule has become one of our simplest quality gates: technical state is observable, evidence state is auditable, and business conclusions wait until the material exists.

Disclosure: this post was drafted with AI assistance and manually reviewed against source notes before publication.

Sources checked July 30, 2026:

• Kickstarter Creator Handbook: https://www.kickstarter.com/help/handbook
• Kickstarter Rules: https://www.kickstarter.com/rules