Copy of Scraping the New York Times with Python and Beautiful Soup

Today we are going to see how we can scrape New York Times articles using Python and BeautifulSoup is a simple and elegant manner.

This article aims to get you started on a real-world problem solving while keeping it super simple, so you get familiar and get practical results as fast as possible.

So the first thing we need is to make sure we have Python 3 installed. If not, you can just get Python 3 and get it installed before you proceed.

Then you can install beautiful soup.
We will also need the library's requests, lxml, and soupsieve to fetch data, break it down to XML, and to use CSS selectors. Install them using.
Once installed, open an editor and type in.

Back to our code now. Let's try and get this data by pretending we are a browser like this.
Save this as nyt_bs.py.

If you run it
You will see the whole HTML page.

Now let's use CSS selectors to get to the data we want. To do that, let's go back to Chrome and open the inspect tool. We now need to get to all the articles. We notice that the

with the class, 'assetWrapper' holds all the individual articles together.
You will see the whole HTML page.

Now let's use CSS selectors to get to the data we want. To do that, let's go back to Chrome and open the inspect tool. We now need to get to all the articles. We notice that the

with the class, 'assetWrapper' holds all the individual articles together.
This selects all the assetWrapper article blocks and runs through them, looking for the element and printing its text.

So when you run it, you get.
Bingo!! We got the article titles.

Now with the same process, we get the class names of all the other data like article link and article summary.
That, when run, should print everything we need from each article like this.
If you want to use this in production and want to scale to thousands of links, then you will find that you will get IP blocked quickly by the New York Times. In this scenario, using a rotating proxy service to rotate IPs is almost a must.

Otherwise, you tend to get IP blocked a lot by automatic location, usage, and bot detection algorithms.

Our rotating proxy server Proxies API provides a simple API that can solve all IP Blocking problems instantly.

With millions of high speed rotating proxies located all over the world
With our automatic IP rotation
With our automatic User-Agent-String rotation (which simulates requests from different, valid web browsers and web browser versions)
With our automatic CAPTCHA solving technology
Hundreds of our customers have successfully solved the headache of IP blocks with a simple API.

A simple API can access the whole thing like below in any programming language.
Now let's go to the NYT home page and inspect the data we can get.

This is how it looks.