DEV Community

Nico Reyes
Nico Reyes

Posted on

Website changed a div class. Scraper returned nothing.

#webscraping #python #tutorial #webdev

Spent 3 weeks building a scraper that pulled job postings from a niche job board. Worked perfectly. Ran it daily, collected data, built a dashboard on top of it.

Then one Tuesday morning the script finished in 4 seconds instead of 2 minutes. Zero results.

Website redesigned their listings page. Changed .job-card to .listing-item. Beautiful Soup kept looking for the old class. Found nothing. Script logged success and moved on like an idiot.

No error. No warning. Just silence.

Thought I could just update the selector:

# Before
jobs = soup.find_all('div', class_='job-card')

# After  
jobs = soup.find_all('div', class_='listing-item')
Enter fullscreen mode Exit fullscreen mode

Worked for 2 days. They changed it AGAIN to .post-container.

I was not doing this every week.

Ended up making it more flexible

Instead of hardcoding class names I targeted the parent container that stayed consistent:

from bs4 import BeautifulSoup
import requests

response = requests.get('https://example-job-board.com/listings')
soup = BeautifulSoup(response.text, 'html.parser')

# Find the main listings container (this never changed)
listings_wrapper = soup.find('div', {'id': 'job-listings'})

if listings_wrapper:
    # Get all direct children (job cards)
    jobs = listings_wrapper.find_all('div', recursive=False)

    for job in jobs:
        # Extract by structure not class names
        title = job.find('h3')  # Title always in h3
        company = job.find('span')  # Company always first span
        link = job.find('a')  # Link always first anchor

        if title and company and link:
            print(f"{title.text.strip()} at {company.text.strip()}")
            print(f"Link: {link.get('href')}\n")
Enter fullscreen mode Exit fullscreen mode

Not bulletproof. But IDs stay stable longer than classes. HTML structure (h3 for title, first span for company) changes less often.

Sites that completely restructure their HTML still break it. Now I check the scraper output daily for sudden drops (50 jobs yesterday, 0 today means something broke).

Thought about switching to ParseForge scrapers since they handle selector changes automatically. Havent done it yet tho you still gotta verify output yourself anyway.

Top comments (0)