Is it ethical to crawl Google Search? Cause they have disallowed all user-agents for scrapping /search path in robots.txt.
/search
Generally speaking, it's fine as long as you don't scrape for a lot of data. That's why I put the warning at the beginning of this post.
Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink.
Hide child comments as well
Confirm
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Is it ethical to crawl Google Search? Cause they have disallowed all user-agents for scrapping
/searchpath in robots.txt.Generally speaking, it's fine as long as you don't scrape for a lot of data. That's why I put the warning at the beginning of this post.