How to crawl website using #bash script? powerexploit on June 27, 2019 Bash script is one of the amazing scripting language used to automate tasks in Linux & Unix and it is one of my favourite scripting language fo... Read full post Submit Preview Dismiss Collapse Expand Sm0ke Sm0ke Sm0ke Follow #Automation, my favorite programming language Location Constanta, Romania Education Hack Land Work Founder @ AppSeed.us Joined Apr 4, 2019 • Jun 27 '19 Dropdown menu Copy link Hide Crawling means to grab a page and extract page data into a structured format. Wget does the first part, download the page. For the second phase, you can use Scrapy or BeautifulSoup Collapse Expand Mohammed Samgan Khan Mohammed Samgan Khan Mohammed Samgan Khan Follow Love to code with a DRY KISS Email mohdsamgankhan@gmail.com Location USA Education B. Tech in CS Work Freelancer Joined Apr 23, 2019 • Jun 27 '19 Dropdown menu Copy link Hide that's what I was wondering, wget will only download the page. Crawling means going through the content of the page. Collapse Expand powerexploit powerexploit powerexploit Follow Security Researcher Location 127.0.01 Joined Jun 26, 2019 • Jun 27 '19 Dropdown menu Copy link Hide I admire your response I know its not a pure crawling but if I only want to crawl one page then I will use wget Or to download or crawl whole I will surely some python stuffs Collapse Expand Arissk Arissk Arissk Follow Location Greece Joined Nov 18, 2019 • Nov 19 '19 Dropdown menu Copy link Hide You can also stay in bash using hxselect and other html and xml bash tools Collapse Expand powerexploit powerexploit powerexploit Follow Security Researcher Location 127.0.01 Joined Jun 26, 2019 • Jun 27 '19 Dropdown menu Copy link Hide I admire your response. You are right to crawl i can use some python like u explain Or also use some tools Collapse Expand Vlastimil Pospichal Vlastimil Pospichal Vlastimil Pospichal Follow Joined Jul 2, 2018 • Jun 29 '19 Dropdown menu Copy link Hide use quotation marks around the variable name: wget "$url". what does the line $url do? case wget in or case $wget in or case "$wget" in? There are significant differences. case wget in is always string "wget". Collapse Expand powerexploit powerexploit powerexploit Follow Security Researcher Location 127.0.01 Joined Jun 26, 2019 • Jun 29 '19 Dropdown menu Copy link Hide -> Wget $url will help me to download page and The whole script working very well. -> double will make it string Collapse Expand Shubhamkhapra Shubhamkhapra Shubhamkhapra Follow automation with bash & python scripting, Former forensic science laboratory . Location Delhi Education forensic and programming Joined Jun 27, 2019 • Jun 27 '19 Dropdown menu Copy link Hide Working harder Collapse Expand powerexploit powerexploit powerexploit Follow Security Researcher Location 127.0.01 Joined Jun 26, 2019 • Jun 27 '19 Dropdown menu Copy link Hide Thanks Collapse Expand Sean O'Donnell Sean O'Donnell Sean O'Donnell Follow Grumpy old linux/perl/php/python geek. Love all the new tools, hate all the buzzwords. Work Software and Systems Engineer Joined Jun 21, 2019 • Jun 27 '19 Dropdown menu Copy link Hide scrapy.org/ Collapse Expand powerexploit powerexploit powerexploit Follow Security Researcher Location 127.0.01 Joined Jun 26, 2019 • Jun 27 '19 Dropdown menu Copy link Hide Good suggestion Collapse Expand Vlastimil Pospichal Vlastimil Pospichal Vlastimil Pospichal Follow Joined Jul 2, 2018 • Jun 28 '19 Dropdown menu Copy link Hide An unusable script with too much mistakes. Pure wget is better. Collapse Expand powerexploit powerexploit powerexploit Follow Security Researcher Location 127.0.01 Joined Jun 26, 2019 • Jun 28 '19 Dropdown menu Copy link Hide What kind of mistakes sir if u can explain plz sir I appreciate your comment &its just a fun script.. Collapse Expand Waleed Barakat Waleed Barakat Waleed Barakat Follow Web developer, Web designer, and Founder of Jodyshop.com Email info@jodyshop.com Location Cairo, Egypt Education High School Work Founder at Jodyshop, Inc. Joined Aug 24, 2018 • Jun 28 '19 Dropdown menu Copy link Hide Nice idea, but what is the benefit of using such crawl process? Collapse Expand powerexploit powerexploit powerexploit Follow Security Researcher Location 127.0.01 Joined Jun 26, 2019 • Jun 28 '19 Dropdown menu Copy link Hide Its just a starting sir I enjoyed a lot when I was doing this Code of Conduct • Report abuse
Crawling means to grab a page and extract page data into a structured format.
Wget does the first part, download the page. For the second phase, you can use Scrapy or BeautifulSoup
that's what I was wondering, wget will only download the page. Crawling means going through the content of the page.
I admire your response I know its not a pure crawling but if I only want to crawl one page then I will use wget
Or to download or crawl whole I will surely some python stuffs
You can also stay in bash using hxselect and other html and xml bash tools
I admire your response.
You are right to crawl i can use some python like u explain
Or also use some tools
wget "$url".$urldo?case wget inorcase $wget inorcase "$wget" in? There are significant differences.case wget inis always string "wget".-> Wget $url will help me to download page and The whole script working very well.
-> double will make it string
Working harder
Thanks
scrapy.org/
Good suggestion
An unusable script with too much mistakes. Pure wget is better.
What kind of mistakes sir if u can explain plz sir I
appreciate your comment
&its just a fun script..
Nice idea, but what is the benefit of using such crawl process?
Its just a starting sir I enjoyed a lot when I was doing this