DEV Community: Shahidul Islam

Do you know what web scraping is?

Shahidul Islam — Thu, 21 Jul 2022 17:41:00 +0000

Simply put, web scraping is the way to extract data from the internet in a structured format.

The internet is undoubtedly full of information, but it is merely structured. For any project which needs data, the format is fundamental. Some might need Excel, some require CSV format, and some need JSON format for use as an API. So, for that reason, web scraping is needed.

How does web scraping work?
A scraper will download the HTML of a page on requests and save entire data or some selected data in a file used for the project.

Which scraping tool is the best?
There is no best scraping tool. All tools are required in different situations. Each website is unique and has its challenge when scraping. Depending on the complexity of the scraping project, the tool selection varies. However, some tools are available like #python packages, no-code tools, browser extensions, e.t.c.

IF YOU LIKE THIS POST, PLEASE HIT THE HEART ICON ON LEFT.

Facts about Web Scraping

Shahidul Islam — Mon, 09 May 2022 19:56:47 +0000

Data science has become a crucial technology in today's business.

Web scraping is one of the essential parts of data source for data science. Here is some point on the facts about web scraping.

Web scraping means pulling data from the internet on a database with the help of a bot. If you are just copy-pasting, it won't be called web scraping.
Although our intuitions say that web scraping is illegal, it is legal to scrape publically available data. See the article https://tinyurl.com/2hbrjcwm. Although if the data is behind the login page, it might be unlawful to scrape.
There are two types of ways you can do web scraping. One is by coding, which is free, and the second is by using third party web scraping services. We use both of them equally.
Most common programming language used is python. And the packages that are primarily used are BeautifulSoup, requests, scrapy etc.
The most used tools are ParseHub, OctoParse, Scraper API, etc. They all have unique features and different price points.
Although web scraping is excellent, there are some challenges. Bot access, IP blocking, Captcha, Honeypot traps, Login requirement, and Dynamic content, to name a few, are the challenges that have to be mitigated for smooth scraping.

If you like this writing, please give a follow.

Copywriting is powerful

Shahidul Islam — Sat, 07 May 2022 20:57:21 +0000

A good copywriter can make you angry, sad, happy, laugh, inspire, moreover make you buy stuff with in few 100 words. That is the power of writing.

Upwork jobs and its statistics.

Shahidul Islam — Tue, 03 May 2022 19:56:00 +0000

I was exhausted about the projects on upwork. I could not find my favourite jobs or projects. I would lose my patience every time I look at the job postings. No job is matching my skillsets. I would like to know about all the jobs and its statistics. So, I went for searching for all the skills on upwork. I found that other job skills have some pros and cons.But for you I'm cutting down all the details. The final result is that, 'copywriting' is the winner. It gets the most job postings and get less competetion. So you could go for the job skill easily. Just learn about copywriting from youtube. That's it for today. Thank you.

Joy of writing code in python

Shahidul Islam — Mon, 25 Apr 2022 19:26:04 +0000

Writing code is an awesome feeling, specially python. What coding does is the near magic you saw in cartoons while growing up. It can be used to do a lot of stuffs for you:

Automating tasks
Make web apps
Creating games
Making useful softwares

If you are not yet in love with coding, I think you should change your perspective. It can make life easier for yourself and others. That is also a scope to help others by writing useful codes for them.