How to crawl website using #bash script?

#bash #linux #github

Bash script is one of the amazing scripting language used to automate tasks in Linux & Unix and it is one of my favourite scripting language for automating the tasks.

A few days ago I was searching about how to crawl website page?
After founding lot of stuff in the internet I learnt about 'Wget' tool into linux system.

Wget is a useful for downloading and crawling a website page.

So after this I started writing a bash script for website page crawling.
-> Firstly open up my favourite vim editor

-> Then started writing script with case statement

->As you can see I uses case statements and automated wget tool into a simple bash script and it its a working code..

For more details about bash and automation
visit my github account

Top comments (15)

Sm0ke • Jun 27 '19

Crawling means to grab a page and extract page data into a structured format.

Wget does the first part, download the page. For the second phase, you can use Scrapy or BeautifulSoup

Mohammed Samgan Khan • Jun 27 '19

that's what I was wondering, wget will only download the page. Crawling means going through the content of the page.

powerexploit • Jun 27 '19

I admire your response I know its not a pure crawling but if I only want to crawl one page then I will use wget
Or to download or crawl whole I will surely some python stuffs

Arissk • Nov 19 '19

You can also stay in bash using hxselect and other html and xml bash tools

powerexploit • Jun 27 '19

I admire your response.
You are right to crawl i can use some python like u explain

Or also use some tools

Vlastimil Pospichal • Jun 29 '19

use quotation marks around the variable name: wget "$url".
what does the line $url do?
case wget in or case $wget in or case "$wget" in? There are significant differences.
case wget in is always string "wget".