If Claude and Codex Can Scrape the Web, Why Do We Still Need Web Scraping Tools?
This question has been on my mind lately.
Tools like Claude and ...
For further actions, you may consider blocking this person and/or reporting abuse
My take: an agent is great at deciding what to fetch and weak as the fetch layer at volume. Once you're at hundreds of URLs on a schedule, you want deterministic code handling retries, sessions, pagination and output validation, with the agent calling it as a tool through a documented input schema. The Apify / Web Scraping Club 2026 survey points the same way: 45.8% of respondents use AI in scraping, and the holdouts gave trust in results, cost and unreliable performance on some sites as their reasons. Does Spicrawl expose an output schema that agents can validate against?
Agree. The agent decides what to fetch. The fetching should be simple, predictable code.
To your question: yes. You can send a JSON Schema with a CSS selector for each field. Spicrawl converts values to the right type ("$1,234.56" becomes 1234.56) and checks the result. With
strict: true, a page that doesn't match fails instead of returning half-right data, and it costs 0 credits. Fields with no match are listed inempty_fields.Errors also include a
retryableflag and a wait time, so your code handles retries, not the model.One limit: it doesn't follow pagination. You send the URLs. For large lists, a batch takes up to 10,000 URLs per call.
Nice, failing at zero credits instead of returning half-right data is exactly right. And the retryable flag is a great touch. Thanks for the detailed answer!
Glad it helped, Dan! Spicrawl is free now, so feel free to try it. We'd love your feedback, good or bad.
Thực tế là Claude/Codex chỉ phù hợp cho ad-hoc exploration — vài trang, dữ liệu nhỏ, one-off. Khi scale lên (hàng nghìn trang, chạy định kỳ, cần monitoring, retry logic, handle anti-bot, parse cấu trúc nhất quán), LLM trở nên đắt đỏ, chậm, và không kiểm soát được output schema.
Hai điểm thường bị bỏ qua:
Tóm lại: LLM là research assistant, scraping tool là data engineering infrastructure. Hai món không thay thế lẫn được (site: labagent .tech)