Web Scraping for Beginners: How to Collect Data Politely and Avoid Getting Blocked

OYINKANSOLA BAKARE AMINAT — Mon, 12 Oct 2026 01:10:33 +0000

When people first hear about web scraping, they often think it's some kind of hacking. It isn't. Scraping just means letting a small program read a web page and pick out the parts you want, like prices, headlines, or job titles, so you don't have to copy them by hand.

The part that trips up most newcomers is getting blocked. You run your script a few times and suddenly the site stops answering. That usually happens because the script behaved in a way no normal visitor would. The good news is that most blocks are easy to avoid.

Read the rules before you start

Type the site's address followed by /robots.txt in your browser (for example website.com/robots.txt). This file tells automated tools which pages they may visit. If a page is off limits, leave it alone. It's also worth skimming the site's terms of service.

Check for an API first

Many sites already share their data through an official API. It's cleaner, it's allowed, and it can save you from writing a scraper at all.

Don't rush the site

A script can send hundreds of requests in a few seconds, and to a server that looks like an attack. A pause of a second or two between requests is usually enough to avoid trouble.

Say who you are

Add a User-Agent with a name for your project and a contact email. It's honest, and site owners tend to be more relaxed about visitors who introduce themselves.

A small example in Python

Here's a short script that puts these habits together. You'll need to install two libraries first: pip install requests beautifulsoup4.

import time
import requests
from bs4 import BeautifulSoup

headers = {"User-Agent": "MyLearningBot/1.0 (contact: you@email.com)"}
urls = ["https://example.com/page1", "https://example.com/page2"]

for url in urls:
    response = requests.get(url, headers=headers, timeout=10)
    if response.status_code == 200:
        soup = BeautifulSoup(response.text, "html.parser")
        print(soup.title.text)
    time.sleep(2)  # wait 2 seconds between requests

What to do if you get a 429 error

A 429 means "too many requests." Stop, wait a while, and slow your script down. Pushing through usually makes things worse.

A quick word on proxies

You'll see proxies mentioned a lot in scraping communities. They send your requests through different addresses, and they have real uses in large, legitimate projects. But they are not a way to ignore a site's rules, and a beginner doesn't need one.

Wrapping up

Stick to public data, keep your speed low, and never collect personal information. Scraping politely keeps you out of trouble and keeps the web usable for everyone.

DEV Community: OYINKANSOLA BAKARE AMINAT