DEV Community

Teddy Zugana
Teddy Zugana

Posted on

1

Python Scraping web page with BeautifulSoup and requests Example

from bs4 import BeautifulSoup
import requests


url = "https://www.point2homes.com/MX/Real-Estate-Listings.html?LocationGeoId=&LocationGeoAreaId=&Location=San%20Felipe,%20Baja%20California,%20Mexico"

headers = {"User-Agent": "Mozilla/5.0","Content-Type": "application/json"}

page_scrape = requests.get(url, headers=headers)
soup = BeautifulSoup(page_scrape.content, 'html.parser')

lists = soup.find_all('article')

for list in lists:
    address = list.find('div', class_="address-container").text
    try:
        beds = list.find('li', class_="ic-beds").text
    except:
        print("Data Not Logged")
    try:
        baths = list.find('li', class_="ic-baths").text
    except:
        print("Data not logged")
    try:
        size = list.find('li', class_="ic-sqft").text
    except:
        print("Data not logged")
    type = list.find('li', class_="property-type ic-proptype").text
    price = list.find('span', class_="green").text
    agent = list.find('div', class_="agent-name").text

    firmstr = list.find('div', class_="agent-company")
    firm=''

    if firmstr is not None:
        spl_word = '>'
        firmstr2=str(firmstr)
        res = firmstr2.split(spl_word, 1)
        splitString = res[1]

        res2 = splitString.split('<', 1)
        splitString2 = res2[0]

        firm=splitString2


    info = [address, beds, baths, size, type, price, agent, firm]

    print(info);
Enter fullscreen mode Exit fullscreen mode

Image of Datadog

How to Diagram Your Cloud Architecture

Cloud architecture diagrams provide critical visibility into the resources in your environment and how they’re connected. In our latest eBook, AWS Solution Architects Jason Mimick and James Wenzel walk through best practices on how to build effective and professional diagrams.

Download the Free eBook

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more