DEV Community: Alireza Mika

AutoScraper and Flask: Create an API From Any Website in Less Than 5 Minutes

Alireza Mika — Mon, 21 Sep 2020 09:44:01 +0000

In this tutorial, we are going to create our own e-commerce search API with support for both eBay and Etsy without using any external APIs. With the power of AutoScraper and Flask, we are able to achieve this goal in fewer than 20 lines of Python code for each site. I recommend reading my last article about AutoScraper if you haven’t done so yet.

Requirements

Install the required libraries using pip:

pip install -U autoscraper flask

Let’s Do It

First, we are going to create a smart scraper to fetch data from eBay’s search results page. Let’s say we want to get the title, price, and product link of each item. Using AutoScraper, it would be easily done by just providing some sample data:

Note that if you want to copy and run this code, you may need to update the wanted_list.

Now let’s get the results grouped by scraping rules:

scraper.get_result_similar(url, grouped=True)

From the output, we’ll know which rule corresponds to which data, so we can use it accordingly. Let’s set some aliases based on the output, remove redundant rules, and save the model so we can use it later:

Note that the rule IDs will be different for you if you run the code.

OK, we’ve got eBay covered. Let’s add support for Etsy search results too. We’ll start by building its scraper. This time, we will use wanted_dict instead of wanted_list. It will automatically set aliases for us:

As the links are generated with a unique ID on Etsy each time, we added one sample product ID to the wanted_dict so we can create the link from it. Also, we provided two samples for title and price, as the structure of items on Etsy search result pages is different and we want the scraper to learn them all.

After analyzing the output, let’s keep our desired rules, remove the rest, and save our model:

Now that we have our scrapers ready, we can create our fully functioning API for both sites in fewer than 40 lines:

Here, we are defining an API with the parameter q as our search query. We will get and join eBay and Etsy search results and return them as a response. Note that we are passing group_by_alias=True to the scraper to get the results grouped by our defined aliases.

By running this code, the API server will be up listening on port 8080. So let’s test our API by opening http://localhost:8080/?q=headphone in our browser:

Some results from eBay

Some results from Etsy

Voila! We have our e-commerce API ready. Just replace headphone in the URL with your desired query to get its search results.

Final Notes

The final code for this tutorial is available on GitHub.

This is a development setup suitable for developing and testing. Flask’s built-in server is not suitable for production. For production usage, please check Flask’s deployment options.

This tutorial is intended for personal and educational use. If you want to scrape websites, you can check their policies regarding the scraping bots.

I hope this article is useful and helps to bring your ideas into code faster than ever. Happy coding!

Smart Automatic Web Scraping in Python

Alireza Mika — Tue, 01 Sep 2020 08:16:34 +0000

In the last few years, web scraping has been one of my day to day and frequently needed tasks. I was wondering if I can make it smart and automatic to save lots of time. So I made AutoScraper!

The project code is available on github:
https://github.com/alirezamika/autoscraper/

This project is made for automatic web scraping to make scraping easy.
It gets a url or the html content of a web page and a list of sample data which we want to scrape from that page. This data can be text, url or any html tag value of that page. It learns the scraping rules and returns the similar elements. Then you can use this learned object with new urls to get similar content or the exact same element of those new pages!

Installation

Install latest version from git repository using pip:

$ pip install git+https://github.com/alirezamika/autoscraper.git

How to use

Getting similar results

Say we want to fetch all related post titles in a stackoverflow page:

from autoscraper import AutoScraper

url = 'https://stackoverflow.com/questions/2081586/web-scraping-with-python'

# We can add one or multiple candidates here.
# You can also put urls here to retrieve urls.
wanted_list = ["How to call an external command?"]

scraper = AutoScraper()
result = scraper.build(url, wanted_list)
print(result)

Here's the output:

[
    'How do I merge two dictionaries in a single expression in Python (taking union of dictionaries)?', 
    'How to call an external command?', 
    'What are metaclasses in Python?', 
    'Does Python have a ternary conditional operator?', 
    'How do you remove duplicates from a list whilst preserving order?', 
    'Convert bytes to a string', 
    'How to get line count of a large file cheaply in Python?', 
    "Does Python have a string 'contains' substring method?", 
    'Why is “1000000000000000 in range(1000000000000001)” so fast in Python 3?'
]

Now you can use the scraper object to get related topics of any stackoverflow page:

scraper.get_result_similar('https://stackoverflow.com/questions/606191/convert-bytes-to-a-string')

We can also append a topic from linked topics section to the wanted list to get all related & linked topics! Or if we want to get its urls, we can add one of the urls too.

Getting exact result

Say we want to scrape live stock prices from yahoo finance:

from autoscraper import AutoScraper

url = 'https://finance.yahoo.com/quote/AAPL/'

wanted_list = ["124.81"]

scraper = AutoScraper()

# Here we can also pass html content via the html parameter instead of the url (html=html_content)
result = scraper.build(url, wanted_list)
print(result)

You can also pass any custom requests module parameter. for example you may want to use proxies or custom headers:

proxies = {
    "http": 'http://127.0.0.1:8001',
    "https": 'https://127.0.0.1:8001',
}

result = scraper.build(url, wanted_list, request_args=dict(proxies=proxies))

Now we can get the price of any symbol:

scraper.get_result_exact('https://finance.yahoo.com/quote/MSFT/')

You may want to get other info as well. For example if you want to get market cap too, you can just append it to the wanted list. By using the get_result_exact method, it will retrieve the data as the same exact order in the wanted list.

Saving the model

We can now save the built model to use it later. To save:

# Give it a file path
scraper.save('yahoo-finance')

And to load:

scraper.load('yahoo-finance')

Generating the scraper python code

We can also generate a stand-alone code for the learned scraper to use it anywhere:

code = scraper.generate_python_code()
print(code)

It will print the generated code. There's a class named GeneratedAutoScraper which has the methods get_result_similar and
get_result_exact which you can use. You can also use get_result method to get both.

Thanks

I hope this project is useful for you and save your time, too.
I look forward to hear any feedback or suggestion.