An automation can introduce itself correctly, stay logged in, and still be the worst visitor on a website.
On October 16, Cloudflare described a new system for continuously evaluating Verified bots. Its initial signals include whether traffic follows published robots preferences and adapts after a site returns HTTP 429. The company says it is using the system internally first; public scores and customer-selectable trust thresholds are planned, not something every customer can configure today.
The useful beginner lesson is smaller than the infrastructure announcement: knowing who your software is does not prove it listens.
If you ask AI to build a crawler, research assistant, or scheduled importer, do not stop testing after it downloads the first page. Test the moment the destination changes its answer.
I would give that behavior its own checklist before expanding the queue.
Start with a tiny, fictional job
Imagine a demo app that imports ten documentation pages from a site you are allowed to access.
The app has two workers. Each worker takes a queued page, downloads it, and records a result. You can pause and restart the process.
That is enough machinery to expose an important problem. Worker one can receive a slowdown signal while worker two keeps sending requests. A restart can forget a pause. A stale queue can contain pages that are no longer eligible.
You do not need to test those mistakes against somebody else's production website. Use a local test server or a controlled fixture with synthetic pages and a request log.
If turning the idea into a bounded first project is the hard part, my $3 AI App Builder Starter Prompts are a guided starting point. For this exercise, keep the project to one queue and five observable tests.
Test 1: A disallowed path never reaches the downloader
Create one allowed demo path and one path excluded by the fixture's robots rules. Put both in the queue.
The result you want is not an error after downloading the excluded page. It is no content request to that path, plus a visible reason for skipping it.
RFC 9309 defines the Robots Exclusion Protocol and explicitly separates it from access authorization. A robots file is not a login system or a universal permission slip. Your app still needs an appropriate basis for accessing the destination and must respect its other controls.
Ask AI to use an established parser for the protocol rather than inventing a few string checks. Then inspect the fixture's actual request log.
My acceptance line would be: "The excluded content URL received zero requests, and the queue explains why."
Test 2: One slowdown reaches both workers
Make the fixture return HTTP 429 with a short wait instruction on the first content request.
RFC 6585 defines 429 as rate limiting and permits a Retry-After header. The protocol does not give your app a universal quota; providers can count requests in different ways.
For this small demo, I would choose a conservative destination-wide pause. Both workers consult the same next-eligible time before starting another request. That is a design choice for the example, not a claim that every API uses a whole-site limit.
Now check the log during the pause. A message saying "worker one is waiting" is insufficient if worker two is still busy hitting the same destination.
Also account for requests that were already in flight. Record when each request started so you can distinguish those from newly dispatched work after the pause became known.
Test 3: Restarting does not erase the pause
Stop the demo during its waiting period. Start it again before that period ends.
The queue should retain the destination's next-eligible time. If the app immediately starts over, the pause existed only in one process's memory.
This is where I would ask AI to show the stored state, not just the function containing a sleep call.
A useful test record includes the destination, the feedback received, the resulting pause, and the next request start time. Keep credentials and response bodies out of that record unless they are actually necessary and safe to retain.
Test 4: Changed preferences affect queued work
Leave several demo pages waiting, then change the fixture's policy before the next batch begins.
Define when your app refreshes that policy and how queued work is checked against the refreshed result. Use the protocol's caching rules and the destination's documented requirements; do not assume one lookup at installation lasts forever.
The important product decision is what happens to old work. I would preserve an excluded item as skipped with its reason, rather than silently deleting it or trying an alternate identity.
Do not route around a refusal with a different account, address, or browser. A technically successful download is not a successful test if it evades the boundary you were meant to honor.
Test 5: The user sees incomplete work honestly
Pause the demo after four of ten pages finish.
The interface should not say "Research complete" because the process exited normally. Show four completed pages, the remaining items, the destination pause, and whether any result is stale or unavailable.
This is also where you decide whether partial output is useful. A documentation search may show the pages it has, with clear coverage. A comparison that requires all ten sources may need to remain incomplete.
Neither decision should be hidden inside the downloader.
A prompt that asks for behavior, not reassurance
I would give AI a prompt like this:
Build a local-only test harness for our documentation importer. Use synthetic pages and no external requests. Add fixtures for an excluded path, HTTP 429 feedback, two workers sharing a destination pause, restart during the pause, and a policy change while work is queued. State each expected request sequence before implementing it. Record request start times and queue transitions. Prove which requests did not occur. Show incomplete coverage truthfully. Do not change identities or bypass controls to make a test pass.
Notice what the prompt does not ask for: "Make the crawler robust."
That adjective leaves too much room for interpretation. A request sequence, a persistent pause, and a visible unfinished queue give you something to inspect.
Keep the first implementation modest
A destination-wide pause can be overly cautious for an API with separate documented limits. A multi-machine deployment needs stronger coordination than a tiny local demo. A fixture cannot prove that an outside service will behave exactly like your simulation.
Those are reasons to state the limits of the test, not reasons to skip it.
Start with one destination and one explicit policy. When you add workers or destinations, repeat the tests at that boundary.
The $3 Starter Prompts help you begin a bounded project. For the organized path through planning, implementation, QA, and publication, AI App Builder From Zero is available as a $9 standalone e-book. The separate 40-prompt workbook is not included. The $10 learning bundle includes the complete book and all 40 Starter Prompts.
Eligible purchases carry one future Bootcamp Bundle benefit: $20 with the $3 prompts, $30 with the standalone book, or $70 with the learning bundle. Verified Apple Books and Kindle combined editions qualify for $70; each store displays its own price. One verified benefit per buyer, used once on either package when checkout opens; benefits do not stack, have no cash value, and exclude refunded or disputed purchases. Previously issued benefits retain their original terms.
Bootcamp Bundle (Review Radar) is coming soon, planned at $399 for one app project or $999 for three. Its planned research, specifications, screen designs, and guided curriculum are not a finished app. You can request waitlist updates through the existing page; checkout remains closed.
The final check is simple: when the destination changes its answer, can you show that every worker changed its behavior?
You can also find me here:
Medium: https://medium.com/@marcusykim
DEV.to: https://dev.to/marcusykim
Website: https://marcusykim.com/
X: https://x.com/marcusykim
LinkedIn: https://www.linkedin.com/in/marcusykim/
Upwork: https://www.upwork.com/freelancers/marcusykim
Top comments (0)