DEV Community: Web Scraping Community

Extract Google Search Results Using Python and BeautifulSoup

Web Scraping Community — Wed, 24 Jan 2024 08:13:06 +0000

You can also read this article on our blog:
Extract Google Search Results Using Python and BeautifulSoup

Web scraping, the process of extracting data from websites, can be a powerful tool for gathering information from the internet.

In this article, we'll explore how to scrape Google search results using Python, BeautifulSoup, and other tools. We'll break down a specific code example and discuss crafting effective selectors for web scraping.

Finally, we'll address the limitations and challenges of such projects.

Setting Up the Environment

Before diving into the code, ensure you have Python installed on your system. Additionally, you'll need to install a few libraries:

Requests: To make HTTP requests in Python.
BeautifulSoup: For parsing HTML and extracting the data.
Rich (optional): For pretty-printing the results.

You can install these libraries using pip:



pip install requests beautifulsoup4 rich

The Code Explained

The provided Python script is structured to extract and display Google search results. Let's go through it step by step:

1. Importing Libraries



import requests
from bs4 import BeautifulSoup
from rich import print
from urllib.parse import urlparse
from urllib.parse import parse_qs

This section imports the necessary modules. requests is used for making HTTP requests, BeautifulSoup for parsing HTML, and rich for enhanced printing. urlparse and parse_qs from urllib.parse are used for URL parsing.

2. Making the Request and Parsing the HTML



query = "Python programming"
url = f"https://www.google.com/search?q={query}"
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
print(soup)

Here, the script constructs a Google search URL for a given query, makes an HTTP request to that URL, and then parses the response using BeautifulSoup.

3. Extracting Google links

The next step involves accurately identifying the various sections of Google's webpage. We are focusing on locating these specific parts:

Thus, for each section, we can extract the specific data we need. In my implementation, the extract_section function handles the parsing of these individual sections.



def extract_results(soup):
    main = soup.select_one("#main")

    res = []
    for gdiv in main.select('.g, .fP1Qef'):
        res.append(extract_section(gdiv))
    return res

4. Extracting each section data

Then, we're looking to extract data from each section, specifically title, link and description:

The key is to check for presence or not of every elements you're pulling data from. This avoid exceptions, and prevent crash on the program during the extraction process.



def extract_section(gdiv):
    # Getting our elements
    title = gdiv.select_one('h3')
    link = gdiv.select_one('a')
    description = gdiv.find('.BNeawe')
    return {
        # Extract title's text only if text is found 
        'title': title.text if title else None,

        'link': link['href'] if link else None,
        'description': description.text if description else None
    }

Let's run it and boom:



[
    {
        'title': 'Welcome to Python.org',
        'link': 
'/url?q=https://www.python.org/&sa=U&ved=2ahUKEwj4wL76qPWDAxVEia8BHWCpAZEQFnoECAIQAg&usg=AOvVaw0ftcoYNT39iYF9FN9-DDSp',
        'description': None
    },
    {
        'title': 'Introduction to Python - W3Schools',
        'link': 
'/url?q=https://www.w3schools.com/python/python_intro.asp&sa=U&ved=2ahUKEwj4wL76qPWDAxVEia8BHWCpAZEQFnoECAYQAg&usg=AOvV
aw1Y76DoERJLKhPAer6y6af0',
        'description': None
    },
    {
        'title': 'Learn Python Programming - Programiz',
        'link': 
'/url?q=https://www.programiz.com/python-programming&sa=U&ved=2ahUKEwj4wL76qPWDAxVEia8BHWCpAZEQFnoECAUQAg&usg=AOvVaw0fo
dl3yhXKlVH6jJnJge3j',
        'description': None
    },
    ...
]

5. Process links

Our links are still not exactly right, they are contained in an q parameter of google's url page. Let's implement extraction for that:



def extract_href(href):
    url = urlparse(href)
    query = parse_qs(url.query)
    if not ('q' in query and query['q'] and len(query['q']) > 0):
        return None
    return query['q'][0]

And add it in our extract_section:



def extract_section(gdiv):
    ...
    return {
        ...
        'link': extract_href(link['href']) if link else None,
        ...
    }

Re-run it and voila!



[

    {'title': 'Welcome to Python.org', 'link': 'https://www.python.org/', 'description': None},

    {

        'title': 'Introduction to Python - W3Schools',

        'link': 'https://www.w3schools.com/python/python_intro.asp',

        'description': None

    },

    {

        'title': 'Learn Python Programming - Programiz',

        'link': 'https://www.programiz.com/python-programming',

        'description': None

    },

    {'title': 'Python Programming Tutorials', 'link': 'https://pythonprogramming.net/', 'description': None},

    ...

]

Crafting Good Selectors

Selectors are crucial in web scraping. They allow you to target specific elements in the HTML document. This script uses CSS selectors like #main, .g, and .fP1Qef to identify parts of the Google search results page.

These selectors are prone to change if Google updates its HTML structure.

The key for good selectors is to keep them simple, the most specific they are the more prone they are to break on even very slight change.

Write selectors that are parent dependent like #main > .fP1Qef only when it's absolutely needed.

What's next?

1. Maintain for changes in the page structure

Google frequently update their HTML structure. This means that a scraper can break easily and requires regular maintenance. Next step for you, is to maintain those selectors when something break.

2. Add proxies

Frequent requests to a website from the same IP can lead to your IP being blocked. Next step is to implement proxies to your code, allowing you to support multiple regions.

3. Data Quality and Reliability

Scraped data might not always be reliable or accurate. You should always verify and validate the data obtained through web scraping, by implementing extensive tests of your code, and testing your implementation on numerous sample pages.

Conclusion

You are now able to integrate Google's search results in your project!

If you want to learn more about Web Scraping & Data extraction and discuss about this article, join us on Discord!

https://discord.com/invite/fHbbHTq4CQ

How to bypass Akamai

Web Scraping Community — Thu, 11 Jan 2024 12:08:23 +0000

Find out more about anti-bots on webscraping.wiki:
How to bypass Akamai and Discord

Akamai is renowned for its advanced anti-bot and anti-scraping solutions, widely adopted by major websites like Nike.com. The effectiveness of Akamai lies in its sophisticated approach that combines traditional fingerprinting techniques with behavioral analysis. This analysis involves tracking the JavaScript events that are triggered during a user's interaction with the website.

Recognizing Akamai

To identify Akamai's presence on a website, look for these characteristics:

Endpoint Structure: Akamai typically operates without external servers, sending data to an endpoint on the website's domain. These endpoints are characterized by long, random character paths that change with every new page load. Each endpoint typically ends with a seven-character string, comprising a mix of random lowercase and uppercase letters. Example from Nike.com:

https://www.nike.com/-V9sK5EuHw5zf/s5azPS0VO0/SGj4/3z9ifbmQLb/M0lvcEBEAQ/a0tSI/UgeeikB

Sensor Data: Akamai sends a key JSON payload named sensor_data to these endpoints. This data is encoded or encrypted using a proprietary algorithm, making it distinct and recognizable. Example payload from Nike.com:

{"sensor_data":"2;4337974;4473908;15,0,0,0,2,0;E9_ 4*o9=r]a0cpl4wxD!z/b4xhdXjOQD4`,=D{bOJjj4:Ob&oj?(6D UK8,-@2;{*U1@Y2jjl+*Uu8=cbeB#HC$]^HeXmd%sZFm5K3rwzDK{=iw;HV}d;J+hn]wr43Y6F}[Q,TF&~?v*V3:q~+jbv7&jh])6JT^wcQF](wf|vPkOrG_(3QG>],jMSj/!Na*w>rh3]WAcv8YMnC) IJJBZKF8E9iPXh2)gmsEmv`@QSY?d:%0KgTC&R9Bj3ZqGX7>=9f2rjbf=X>.SExZq45>i1FGWkIf-Cl8]BL2ffk2g1l_8%aG^=5: Q[ct?Y*m(f%MFU}hlxNq[68ejD&W~27%jru9N5[>bJ[5;B~f|~i!sU^54s_0g}$w[_fDt<*R-v3by%gUu#d1w`Q52K2Y=`z[KevKiU*#X8X9HWG2hnUd`{q`jR[%q$M@$Si;l^.{iR$Sx&(~-W;z}l%q)lpP6Q}pd(E/RF+}dO8n,@M.n5|MhZgk}euOJqC5=^..."}

Cookies: Akamai set the following cookies on the client browser:

_abck
ak_bmsc
bm_mi
bm_sz
bm_sv

Akamai's Device Fingerprinting

Akamai extensively uses techniques like Canvas Fingerprinting and WebGL fingerprinting to identify user devices. This is achieved using JavaScript, as illustrated in the provided scripts. These scripts create a unique identifier for each device based on how it renders certain elements.

var n = document.createElement("canvas");
if (n.width = 280, n.height = 60, n.style.display = "none", "function" == typeof n.getContext) {
  var o = n.getContext("2d");
  o.fillStyle = "rgb(102, 204, 0)",
    o.fillRect(100, 5, 80, 50),
    o.fillStyle = "#f60",
    o.font = "16pt Arial",
    o.fillText(t, 10, 40),
    o.strokeStyle = "rgb(120, 186, 176)",
    o.arc(80, 10, 20, 0, Math.PI, !1),
    o.stroke();
  var m = n.toDataURL();
  e = 0;
  for (var r = 0; r < m.length; r++) {
    e = (e << 5) - e + m.charCodeAt(r),
    e &= e
  }
  e = e.toString();
  var i = document.createElement("canvas");
  i.width = 16,
    i.height = 16;
  var c = i.getContext("2d");
  c.font = "6pt Arial",
    a.rVal = Math.floor(1e3 * Math.random()).toString(),
    c.fillText(a.rVal, 1, 12);
  for (var b = i.toDataURL(), d = 0, s = 0; s < b.length; s++) {
    d = (d << 5) - d + b.charCodeAt(s),
      d &= d
  }
  a.rCFP = d.toString()
}

try {
  var t = document.createElement("canvas"),
    a = t.getContext("webgl");
      bmak.wv = "n",
      bmak.wr = "n",
      bmak.weh = "n",
      bmak.wl = 0,
    a && (bmak.wv = "b",
      bmak.wr = "b",
      bmak.weh = "b",
    a.getSupportedExtensions() && (bmak.weh = bmak.ats(bmak.mn_s(JSON.stringify(a.getSupportedExtensions().sort()))),
      bmak.wl = a.getSupportedExtensions().length,
      a.getSupportedExtensions().indexOf("WEBGL_debug_renderer_info") >= 0 && (bmak.wv = a.getParameter(a.getExtension("WEBGL_debug_renderer_info").UNMASKED_VENDOR_WEBGL),
      bmak.wr = a.getParameter(a.getExtension("WEBGL_debug_renderer_info").UNMASKED_RENDERER_WEBGL))))
} catch (t) {
   bmak.wv = "e",
   bmak.wr = "e",
   bmak.weh = "e",
   bmak.wl = 0
}

Akamai Behavioral Analysis

In its highest security settings, Akamai can capture multiple sensor_data payloads per minute. This includes recording various user interactions like mouse movements, clicks, touch events, and keyboard inputs. The system also tracks the positions of these events on the page.

Here is the event it is listening to:

touchmove
touchstart
touchend
touchcancel
mousemove
click
mousedown
mouseup
pointerdown
pointerup
keydown
keyup
keypress
touchmove
touchstart
touchend
touchcancel
onmousemove
onclick
onmousedown
onmouseup
onpointerdown
onpointerup
onkeydown
onkeyup
onkeypress

Bypassing Akamai

Bypassing Akamai's security requires a nuanced approach, especially considering its reliance on GPU rendering information to determine the operating system and device type.

Here are some strategies:

GPU Rendering: Emulate consumer-grade GPUs rather than professional hardware, as Akamai's algorithms are tuned to recognize and differentiate between them.
Behavioral Analysis: Utilize tools like ghost-cursor (found at https://npmjs.com/package/ghost-cursor) to simulate human-like cursor movements and keystrokes. Timing is crucial here; movements or keystrokes that are too rapid can be flagged as suspicious.