DEV Community: Max Klein

How spreading activation beats keyword search for LLM agents

Max Klein — Thu, 12 Mar 2026 19:51:57 +0000

LLM agents are powerful reasoners but poor navigators.

They can analyze code, write solutions, and debug problems. But ask them "what does this change affect?" or "what am I missing?" and they're stuck grepping through flat files.

I built m1nd to fix this.

The idea

Instead of searching your data, m1nd activates it. You query a concept, and the graph lights up: connected nodes fire with decaying signal, noise gets cancelled, and the system learns from your feedback.

You ask:  "What's related to authentication?"
m1nd:     Activates auth -> propagates to session, middleware, JWT
          -> detects structural hole (no rate limiter connected)
          -> predicts co-change with password reset module
          -> all in one query, ranked by multi-dimensional relevance

How it works

1. Ingest - Build a property graph from your data (code, JSON, any domain)

2. Activate - Spreading activation across 4 dimensions:

Structural: graph topology, PageRank, community structure
Semantic: label similarity via char n-grams and co-occurrence
Temporal: recency decay, change velocity, co-change history
Causal: directed dependency flow along import/call/contain edges

3. Cancel noise - XLR differential processing maintains parallel signal and noise graphs, gating activation through an adaptive sigmoid. Borrowed from balanced audio engineering.

4. Learn - Hebbian plasticity: tell m1nd "this result was correct" and it strengthens the connections (LTP). "This was wrong" weakens them (LTD). Over time, the graph gets smarter.

5. Persist - Graph + plasticity state saves to disk. Next session starts where this one left off.

The 13 tools

m1nd exposes 13 tools over MCP (Model Context Protocol), callable by any LLM agent:

Category	Tools
Discovery	activate, why, missing, fingerprint, resonate
Change Analysis	impact, predict, counterfactual
Learning	learn, drift, warmup
System	ingest, health

Real example

m1nd analyzing its own codebase (693 nodes, 2007 edges):

> m1nd.activate query="spreading activation"
0.94 fn::propagate_wavefront  (structural 0.97)
0.91 fn::score_candidates     (semantic 0.93)
0.87 fn::xlr_gate             (causal 0.89)
XLR cancelled 12 noise edges

> m1nd.counterfactual remove=["graph.rs"]
CASCADE: 31 nodes lose connectivity
ORPHANED: 8 functions become unreachable
graph.rs is load-bearing

Why not vector search?

Vector search finds things that look similar. Spreading activation finds things that are structurally connected. The difference matters:

Blast radius requires graph topology, not cosine similarity
Structural holes are invisible to embeddings
Co-change prediction needs temporal graph history
Counterfactual simulation is impossible without structure
Hebbian learning is online and incremental - no retraining

Try it

Built in Rust. 15.5K LOC. 159 tests. 3.8MB binary.

cargo build --release
./target/release/m1nd-mcp

GitHub: github.com/maxkle1nz/m1nd
Landing: maxkle1nz.github.io/m1nd
Integration guide: INTEGRATION-GUIDE.md

*Max Kleinschmidt *

Building a Stock Price Bot with Python: Complete Tutorial

Max Klein — Mon, 02 Mar 2026 14:10:25 +0000

In today’s fast-paced financial markets, staying ahead of the curve means having access to real-time data and automation. Imagine a tool that continuously monitors stock prices, sends alerts when thresholds are crossed, and even logs historical data for analysis—all without lifting a finger. That’s the power of a stock price bot, and in this tutorial, you’ll learn how to build one using Python. Whether you’re a developer looking to automate your trading strategies or a beginner eager to dive into financial data, this guide will walk you through every step of the process. Let’s get started!

Prerequisites

Before we jump into coding, ensure your environment meets the following requirements:

📦 Software and Tools

Python 3.8+ installed on your system. Download Python here.
pip (Python’s package installer) for managing dependencies.
A code editor like Visual Studio Code, PyCharm, or Jupyter Notebook.

📌 Required Libraries

You’ll need the following Python libraries:

yfinance: For fetching stock data from Yahoo Finance.
requests: For interacting with APIs (if needed).
schedule: For scheduling recurring tasks (e.g., daily price checks).
datetime: For handling dates and times.

Install them using pip:

pip install yfinance requests schedule

Tip: Always use a virtual environment (via venv or conda) to isolate dependencies and avoid conflicts.

Setting Up Your Development Environment

Create a Project Folder Organize your code by creating a dedicated directory for your bot:

   mkdir stock_price_bot
   cd stock_price_bot

Initialize a Virtual Environment

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

Install Dependencies Run the pip install command from the prerequisites section.

Step 1: Fetching Real-Time Stock Data with yfinance

The yfinance library is a game-changer for stock data. It provides access to Yahoo Finance’s API, allowing you to retrieve current prices, historical data, and company fundamentals with minimal code.

✅ Example: Get the Current Price of a Stock

import yfinance as yf

def get_current_price(symbol):
    try:
        stock = yf.Ticker(symbol)
        price = stock.info["regularMarketPrice"]
        return price
    except Exception as e:
        print(f"Error fetching data for {symbol}: {e}")
        return None

# Example usage
symbol = "AAPL"
price = get_current_price(symbol)
if price:
    print(f"Current price of {symbol}: ${price:.2f}")

🔍 Explanation

yf.Ticker(symbol) initializes a stock object.
stock.info is a dictionary containing metadata like price, volume, and market cap.
regularMarketPrice gives the latest price (may vary slightly based on market hours).

Warning: Yahoo Finance’s API may throttle requests if you make too many calls in a short time. Use this responsibly.

Step 2: Monitoring Price Changes and Sending Alerts

Now that we can fetch current prices, let’s build a function to monitor changes and send alerts when a stock crosses a predefined threshold.

🚨 Example: Price Alert System

import time
import smtplib  # For sending emails (basic example)

def send_email_alert(subject, body, to_email):
    from_email = "your_email@example.com"
    from_password = "your_email_password"

    try:
        server = smtplib.SMTP("smtp.gmail.com", 587)
        server.starttls()
        server.login(from_email, from_password)
        message = f"Subject: {subject}\n\n{body}"
        server.sendmail(from_email, to_email, message)
        server.quit()
        print("Email alert sent successfully!")
    except Exception as e:
        print(f"Failed to send email: {e}")

def monitor_price(symbol, threshold, email):
    while True:
        price = get_current_price(symbol)
        if price and price > threshold:
            send_email_alert(
                subject=f"Price Alert for {symbol}",
                body=f"The price of {symbol} has exceeded ${threshold}! Current price: ${price:.2f}",
                to_email=email
            )
        time.sleep(60)  # Check every 60 seconds

# Example usage
monitor_price("TSLA", 300, "your_email@example.com")

⚠️ Important Notes

Replace your_email@example.com and your_email_password with valid credentials. Consider using OAuth2 or email services like Mailgun for better security.
This is a basic example. In production, avoid hardcoding sensitive info like passwords.

Step 3: Logging Historical Data for Analysis

Tracking historical data lets you analyze trends and make data-driven decisions. Let’s modify our bot to log prices to a CSV file.

📊 Example: Save Historical Data

import csv
from datetime import datetime

def log_price_to_csv(symbol, price):
    filename = f"{symbol}_history.csv"
    try:
        with open(filename, "a", newline="") as file:
            writer = csv.writer(file)
            writer.writerow([datetime.now(), price])
        print(f"Logged price for {symbol} to {filename}")
    except Exception as e:
        print(f"Error logging data: {e}")

# Example usage
log_price_to_csv("AAPL", 190.50)

📈 Enhancements

Use Pandas to handle CSV files for more complex analysis.
Store data in a database (e.g., SQLite) for scalability.

Step 4: Automating the Bot with a Scheduler

Manually running the bot isn’t efficient. Let’s use the schedule library to automate checks at regular intervals.

⏰ Example: Schedule Daily Price Checks

import schedule
import time

def job():
    print("Running scheduled task...")
    monitor_price("AAPL", 190, "your_email@example.com")

# Schedule the job every day at 9 AM
schedule.every().day.at("09:00").do(job)

while True:
    schedule.run_pending()
    time.sleep(1)

Tip: For production use, consider using cron jobs (Linux/Mac) or Task Scheduler (Windows) for more reliable automation.

Step 5: Deploying Your Bot

Once your bot is working locally, deploy it to a cloud service or server for continuous operation.

🚀 Deployment Options

Heroku: Easy to deploy Python apps with a free tier.
AWS Lambda: Serverless option for running scripts on a schedule.
DigitalOcean: Affordable VPS for full control.

🛡️ Best Practices for Deployment

Use environment variables to store API keys and credentials.
Set up monitoring (e.g., via Sentry or Datadog) to track errors.
Implement rate limiting to avoid API abuse.

Conclusion

Congratulations—you’ve built a fully functional stock price bot that fetches real-time data, sends alerts, and logs historical information! This is just the beginning. With Python’s versatility, you can expand your bot to include features like:

GUI dashboards (using Tkinter or Streamlit).
Machine learning models for price prediction.
Integration with trading platforms (e.g., Alpaca or Interactive Brokers).

Next Steps

Here are some ideas to take your bot to the next level:

Add Multiple Stock Support

Modify the code to handle a list of symbols and thresholds.
Implement Webhooks

Use services like IFTTT or Zapier to trigger actions based on price changes.
Build a Web Interface

Use Flask or Django to create a dashboard for viewing stock data.
Explore Alternative APIs

Try Alpha Vantage, IEX Cloud, or Polygon.io for more advanced features.
Automate Backtesting

Use historical data to simulate trading strategies and evaluate performance.

Final Thoughts

Building a stock price bot is a powerful way to automate financial monitoring and analysis. As you continue refining your bot, always prioritize security, reliability, and scalability. Whether you’re tracking your portfolio or experimenting with algorithmic trading, Python gives you the tools to succeed. Now go forth and code! 🚀

Built by N3X1S INTELLIGENCE — we automate data extraction at scale. Need custom scraping, bots, or data pipelines? Hire us on Fiverr.

Rotating Proxies for Web Scraping: Complete Guide

Max Klein — Mon, 02 Mar 2026 13:50:24 +0000

Web scraping is a powerful tool for extracting data from websites, but it comes with a unique set of challenges. One of the most common roadblocks? Getting blocked by websites that detect and ban repeated requests from the same IP address. This is where rotating proxies step in, acting as a shield that lets you scrape data without exposing your real IP or getting locked out. In this guide, we’ll walk you through everything you need to know about rotating proxies for web scraping, from the fundamentals to hands-on Python code examples that actually work.

Whether you’re a developer building a data pipeline or a researcher gathering public information, this article will help you avoid the pitfalls of IP bans, respect website terms of service, and scrape efficiently at scale.

Prerequisites

Before diving into this tutorial, ensure you have the following:

Basic Python knowledge: Familiarity with Python syntax, functions, and libraries like requests and BeautifulSoup.
Understanding of HTTP: A grasp of HTTP methods (GET, POST), headers, and status codes.
Web scraping basics: Experience with parsing HTML or JSON data from websites.
Tools installed: Install Python 3.x and the required libraries using pip:

pip install requests lxml

What Are Rotating Proxies?

Definition and Purpose

A rotating proxy is a service that automatically changes your IP address after a set number of requests or at regular intervals. This prevents websites from detecting and blocking your scraper by making it appear as though multiple users are accessing the site.

Unlike a static proxy, which uses a single IP address, rotating proxies cycle through a pool of IPs, ensuring your scraper remains undetected and your data collection uninterrupted.

Use Cases for Rotating Proxies

Avoiding IP bans on e-commerce or social media platforms.
Scraping large datasets without triggering anti-scraping mechanisms.
Simulating real user behavior by rotating IPs and user agents.

How Rotating Proxies Work

The Mechanics Behind the Scenes

When you use a rotating proxy, your requests pass through an intermediary server before reaching the target website. Here’s a simplified breakdown of the process:

Request Initialization: Your scraper sends an HTTP request to the proxy server.
IP Rotation: The proxy server assigns a new IP address from its pool and forwards the request to the target website.
Response Handling: The proxy receives the website’s response and sends it back to your scraper.

This process happens seamlessly, with no noticeable latency for the user.

Key Components of a Rotating Proxy System

Proxy Pool: A collection of IP addresses managed by the proxy service.
Rotation Logic: Rules that dictate how often or under what conditions IPs are rotated.
Authentication: Most proxy services require API keys or credentials to access their network.

Benefits of Using Rotating Proxies

1. Avoid IP Bans and Account Lockouts

Websites often flag suspicious activity, such as rapid-fire requests from the same IP. Rotating proxies mimic organic traffic, reducing the risk of being blocked.

2. Bypass Rate Limits

Many APIs and websites impose rate limits on requests. Rotating proxies allow you to distribute your load across multiple IPs, staying under these limits.

3. Improve Scraping Efficiency

By using a large proxy pool, you can scrape multiple pages simultaneously without hitting the same IP repeatedly.

4. Enhance Data Accuracy

Some websites serve different content based on the user’s location. Rotating proxies let you scrape data from various regions, ensuring a more comprehensive dataset.

Choosing the Right Rotating Proxy Service

Factors to Consider

Speed and Reliability: Ensure the proxy service has low latency and high uptime.
IP Pool Size: Larger pools reduce the chance of hitting the same IP.
Support for HTTPS: Proxies that support HTTPS are essential for secure scraping.
Pricing Model: Free proxies may be slow or unreliable; paid services often offer better performance and support.
Legal and Ethical Compliance: Choose a provider that respects website terms of service and avoids blacklisted IPs.

Popular Rotating Proxy Providers

BrightData (formerly Luminati)
Oxylabs
Smartproxy
ProxyMesh

Tip: Always test a proxy service with a small script before committing to a paid plan.

Practical Code Examples: Using Rotating Proxies in Python

Example 1: Basic Rotating Proxy with `requests`

Below is a simple script that uses a rotating proxy list. For demonstration purposes, we’ll use a static list of proxies, but in a real-world scenario, you’d fetch these from a proxy service API.

import requests
from itertools import cycle

# Sample list of proxies (replace with your own)
proxies = [
    'http://10.10.1.10:3128',
    'http://10.10.1.11:3128',
    'http://10.10.1.12:3100',
]

# Create a cycle iterator for the proxies
proxy_pool = cycle(proxies)

def get_html(url):
    try:
        # Get the next proxy from the pool
        proxy = next(proxy_pool)
        print(f"Using proxy: {proxy}")

        # Create a session with the proxy
        session = requests.Session()
        session.proxies = {'http': proxy, 'https': proxy}

        # Send a GET request
        response = session.get(url, timeout=10)
        return response.text
    except Exception as e:
        print(f"Error: {e}")
        return None

# Example usage
url = 'https://httpbin.org/ip'
html = get_html(url)
if html:
    print(html)

Warning: Free proxy lists are often unreliable and may include malicious IPs. Always use a trusted rotating proxy service for production work.

Example 2: Rotating Proxies with a Proxy Service API

Most paid proxy services provide an API to fetch proxies dynamically. Here’s an example using BrightData (replace placeholders with your credentials):

import requests
import time

# BrightData API endpoint (example)
api_url = 'https://api.brightdata.com/proxy/v1/get'

# Your BrightData credentials
username = 'your_username'
password = 'your_password'

def get_rotating_proxy():
    try:
        response = requests.get(
            api_url,
            auth=(username, password),
            params={'type': 'http', 'country': 'US'},
            timeout=10
        )
        response.raise_for_status()
        proxy = response.json()['proxy']
        return proxy
    except Exception as e:
        print(f"Failed to fetch proxy: {e}")
        return None

def scrape_with_rotating_proxy(url):
    proxy = get_rotating_proxy()
    if not proxy:
        print("No proxy available.")
        return

    try:
        session = requests.Session()
        session.proxies = {'http': proxy, 'https': proxy}
        response = session.get(url, timeout=10)
        print(f"Response from {url} using proxy {proxy}:")
        print(response.text)
    except Exception as e:
        print(f"Error: {e}")

# Example usage
scrape_with_rotating_proxy('https://httpbin.org/ip')
time.sleep(5)  # Wait before next request
scrape_with_rotating_proxy('https://httpbin.org/ip')

Best Practices for Using Rotating Proxies

1. Rotate Proxies at Appropriate Intervals

Too frequent: May trigger rate limits or suspicious activity.
Too infrequent: Increases risk of IP bans.

Use a balance based on the target website’s tolerance. A common approach is to rotate every 10–100 requests.

2. Use User-Agent Rotation

Even with proxies, websites may block requests based on headers. Rotate user agents to mimic different browsers and devices:

import random

user_agents = [
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Safari/605.1.15',
]

headers = {'User-Agent': random.choice(user_agents)}
response = requests.get(url, headers=headers)

3. Respect Website Policies

Avoid scraping content that violates the robots.txt file.
Implement delays between requests (e.g., time.sleep(2)) to reduce load on servers.
Use headers to mimic a real browser.

Best Practice: Always include a User-Agent and Accept-Language header in your requests.

4. Monitor and Handle Errors

Implement error handling to manage connection issues, timeouts, and failed proxy rotations:

try:
    response = session.get(url, timeout=10)
    response.raise_for_status()
except requests.exceptions.RequestException as e:
    print(f"Request failed: {e}")
    # Optionally, retry with a new proxy

Advanced Techniques: Combining Proxies with Web Scraping Frameworks

Using `Scrapy` with Rotating Proxies

Scrapy is a powerful web scraping framework that integrates well with rotating proxies. Here’s a basic settings.py configuration:

# settings.py
DOWNLOADER_MIDDLEWARES = {
    'scrapy_proxypool.middleware.ProxyPoolMiddleware': 100,
    'scrapy_proxypool.middleware.RotatingProxyMiddleware': 110,
}

PROXYPOOL_ENABLED = True
PROXYPOOL_PROVIDERS = ['scrapy_proxypool.providers.brightdata.BrightData']
PROXYPOOL_BRIGHTDATA_USERNAME = 'your_username'
PROXYPOOL_BRIGHTDATA_PASSWORD = 'your_password'

This setup automatically rotates proxies for every request, making it ideal for large-scale scraping projects.

Conclusion

Rotating proxies are a game-changer for anyone serious about web scraping. They allow you to scrape data at scale without triggering anti-scraping measures, while also improving the reliability and efficiency of your scraper. By understanding how they work, choosing the right service, and implementing best practices, you can build robust, long-lasting scraping solutions.

Next Steps

Now that you’ve learned the basics, consider exploring these advanced topics:

Automating proxy rotation with cron jobs or task queues (e.g., Celery).
Integrating proxy rotation with headless browsers like Selenium or Playwright.
Using CAPTCHA-solving services to bypass security challenges.
Building your own proxy pool with residential IPs.

Remember: Always stay within legal and ethical boundaries when scraping. Respect website owners and use your skills to create value, not harm.

Happy scraping! 🕵️‍♂️

Need professional web scraping done for you? N3X1S INTELLIGENCE on Fiverr delivers clean, structured data from any website. Same-day delivery available.

How to Scrape Google Search Results Legally

Max Klein — Mon, 02 Mar 2026 13:30:00 +0000

In today’s data-driven world, the ability to extract and analyze information from the web is a powerful skill. However, scraping Google search results presents a unique challenge: how to do it without breaking the law or violating Google’s terms of service. Many developers face this dilemma, trying to balance the need for data with the ethical and legal responsibilities that come with it. The good news? It’s entirely possible to scrape Google search results legally if you follow the right approach. In this tutorial, I’ll walk you through the technical and legal steps to achieve this, using Python as our primary tool. By the end, you’ll have working code examples and a deep understanding of best practices to avoid getting blocked, fined, or worse.

Prerequisites

Before diving into the tutorial, ensure you have the following:

Tools and Knowledge Required

Python 3.x: The language we’ll use for all code examples.
Basic Python Skills: Familiarity with libraries like requests, BeautifulSoup, and Selenium.
Google Cloud Account: Required if using the Google Custom Search API (explained later).
Web Scraping Ethics: Understanding of robots.txt, rate limits, and legal boundaries.

Software Installation

Install the necessary Python libraries:

pip install requests beautifulsoup4 selenium google-api-python-client

💡 Tip: Always use a virtual environment to manage dependencies for projects like this.

Legal Considerations: Why You Can’t Just Scrape Google

Before writing any code, it’s critical to understand why scraping Google is a minefield. Google’s Terms of Service explicitly prohibit automated access to its search results without permission. Violating these terms can result in:

Your IP address being blocked.
Legal action from Google.
Reputational damage to your project or company.

So, how do we scrape legally? The answer lies in using Google’s official APIs and respecting their policies. Let’s explore the two primary methods: Google Custom Search API and browser automation with Selenium.

Method 1: Using Google’s Custom Search API (Recommended)

Google’s Custom Search API is the only legal and officially supported way to access search results programmatically. It’s designed for developers who need to integrate Google search functionality into their apps.

Step 1: Set Up a Google Cloud Project

Go to the Google Cloud Console.
Create a new project or select an existing one.
Enable the Custom Search API.
Create a Custom Search Engine (CSE):
- Go to Custom Search Engine.
- Define your search engine’s scope (e.g., Google’s entire index or a specific site).
- Note the Search Engine ID (CX).
Generate an API key under the Google Cloud Console’s APIs & Services > Credentials section.

Step 2: Use the API in Python

Here’s a complete example of how to use the API to fetch search results:

from googleapiclient.discovery import build
import os

# Set up API credentials
API_KEY = "YOUR_API_KEY"
CX = "YOUR_CX"

# Build the service
service = build("customsearch", "v1", developerKey=API_KEY)

# Perform a search
def search_google(query):
    result = service.cse().list(q=query, cx=CX, num=10).execute()
    return result["items"]

# Example usage
results = search_google("how to learn Python")
for item in results:
    print(f"Title: {item['title']}")
    print(f"Link: {item['link']}")
    print(f"Snippet: {item['snippet']}\n")

⚠️ Warning: The Custom Search API has rate limits and costs. Free-tier usage is limited to 100 searches per day. For production use, consider upgrading to a paid plan.

Method 2: Scraping Google Search Results with Selenium (For Advanced Users)

If the API isn’t sufficient (e.g., you need to scrape non-English results or bypass limitations), you can use Selenium to automate a browser. However, this approach requires careful handling to avoid getting blocked.

Step 1: Set Up Selenium

Install Selenium and a WebDriver (e.g., ChromeDriver):

pip install selenium

Download the appropriate WebDriver from https://chromedriver.chromium.org/.

Step 2: Automate Google Search

Here’s a complete example of scraping Google search results using Selenium:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup
import time

# Set up Chrome options for headless mode
chrome_options = Options()
chrome_options.add_argument("--headless")  # Run in background
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-dev-shm-usage")

# Path to your ChromeDriver
driver_path = "/path/to/chromedriver"

# Initialize the browser
service = Service(driver_path)
driver = webdriver.Chrome(service=service, options=chrome_options)

def scrape_google_search(query):
    url = f"https://www.google.com/search?q={query}"
    driver.get(url)

    # Wait for JavaScript to load
    time.sleep(3)

    # Extract results using BeautifulSoup
    html = driver.page_source
    soup = BeautifulSoup(html, "html.parser")

    # Find all search result links
    results = soup.find_all("div", class_="tF2Cxc")
    for result in results:
        title = result.find("h3").text
        link = result.find("a")["href"]
        snippet = result.find("div", class_="VwiC3b").text
        print(f"Title: {title}\nLink: {link}\nSnippet: {snippet}\n")

# Example usage
scrape_google_search("how to learn Python")
driver.quit()

🔒 Tip: Use a proxy rotation service to avoid IP blocking when scraping at scale.

Best Practices for Legal and Ethical Scraping

Even with the right tools, you must follow best practices to avoid issues:

1. Respect Robots.txt

Check Google’s robots.txt file (https://www.google.com/robots.txt) to ensure your scraping activity is allowed.

2. Implement Rate Limiting

Add delays between requests to avoid overwhelming Google’s servers:

import time

def scrape_google_search(query):
    time.sleep(2)  # Wait 2 seconds between requests
    # ... rest of the code

3. Rotate User-Agents

Use different user-agent strings to mimic human browsing:

from fake_useragent import UserAgent

ua = UserAgent()
headers = {"User-Agent": ua.random}

⚠️ Warning: Google may block your IP if it detects automated traffic. Always use this method as a last resort.

4. Handle CAPTCHA and Blocks Gracefully

If you hit a CAPTCHA, pause the scraper and retry later. Use libraries like selenium-wire to manage proxies and bypass CAPTCHA if legal.

Advanced Techniques: Combining APIs and Scraping

For maximum flexibility, consider hybrid approaches:

Use the Custom Search API for structured data.
Use Selenium to scrape content that isn’t available via the API (e.g., rich snippets, images).
Store results in a database for later analysis.

Conclusion

Scraping Google search results legally is possible—but it requires careful planning, adherence to Google’s policies, and the right tools. Whether you use the Custom Search API or browser automation with Selenium, always prioritize ethics, legality, and respect for Google’s infrastructure.

By following the steps outlined in this tutorial, you’ll avoid the pitfalls that trap many developers and build robust, compliant solutions.

Next Steps

Here’s what to do next:

Explore the Google Custom Search API documentation for advanced features like filtering by region or language.
Experiment with headless browser scraping using Selenium and proxy rotation.
Learn about ethical scraping frameworks like Scrapy and Playwright.
Study rate-limiting strategies to avoid getting blocked.
Consider legal and compliance training if you’re building products that rely on web scraping.

Remember: The goal isn’t just to scrape data—it’s to do so responsibly. Stay curious, stay compliant, and keep building.

Need professional web scraping done for you? Check out N3X1S INTELLIGENCE on Fiverr — fast, reliable data extraction from any website.

Python asyncio for Web Scraping: Speed Up 10x

Max Klein — Mon, 02 Mar 2026 13:10:23 +0000

Python asyncio for Web Scraping: Speed Up 10x

Web scraping is a powerful technique for extracting data from websites, but traditional synchronous methods can be painfully slow when dealing with large-scale or high-latency targets. What if you could scrape 10 times faster with the same code? That’s where Python’s asyncio library shines. By leveraging asynchronous I/O and non-blocking network requests, asyncio can transform your web scraping workflows from sluggish to lightning-fast.

In this tutorial, we’ll walk you through the fundamentals of using asyncio for web scraping, from setting up your environment to writing high-performance scrapers that scale effortlessly. Whether you’re a seasoned developer or just dipping your toes into asynchronous programming, this guide will give you the tools to boost your scraping speed and efficiency by orders of magnitude.

Understanding AsyncIO and Its Role in Web Scraping

What is AsyncIO?

asyncio is a Python library that enables asynchronous, non-blocking code through coroutines and event loops. It allows your program to perform multiple I/O-bound tasks concurrently without waiting for each one to complete sequentially.

Why AsyncIO for Web Scraping?

Traditional web scraping with requests sends one request at a time, blocking the program until it receives a response. With asyncio, you can send dozens or even hundreds of requests simultaneously, dramatically reducing total execution time.

Real-World Impact

A study by a major e-commerce platform showed that switching from synchronous to asynchronous scraping reduced their data collection time from 10 minutes to under a minute for the same dataset. That’s a 10x speedup—and it’s achievable with minimal code changes.

Step 2: Define an Async Function for Scraping

async def fetch_url(session, url):
    try:
        async with session.get(url, timeout=10) as response:
            if response.status == 200:
                html = await response.text()
                return html
            else:
                print(f"Failed to fetch {url}: Status code {response.status}")
                return None
    except Exception as e:
        print(f"Error fetching {url}: {e}")
        return None

Warning: Always include try/except blocks to handle network errors gracefully. Use timeout to avoid hanging indefinitely on slow or unresponsive servers.

Extracting Data from HTML with BeautifulSoup

Step 4: Parse HTML and Extract Data

Let’s modify the fetch_url function to parse HTML and extract specific data:

async def fetch_and_parse(session, url):
    try:
        async with session.get(url, timeout=10) as response:
            if response.status == 200:
                html = await response.text()
                soup = BeautifulSoup(html, "html.parser")
                # Example: Extract all <a> tags
                links = [a["href"] for a in soup.find_all("a", href=True)]
                return {"url": url, "links": links}
            else:
                print(f"Failed to fetch {url}: Status code {response.status}")
                return None
    except Exception as e:
        print(f"Error fetching {url}: {e}")
        return None

Example Output

{
    "url": "https://example.com/page1",
    "links": [
        "/about", "/contact", "/products", ...
    ]
}

Tip: Use soup.find_all() with specific tags and attributes to target only the data you need. Avoid parsing the entire HTML if you can narrow your focus.

Handling Complex Scenarios: Pagination and JavaScript

Step 6: Scrape Paginated Pages

If a site has paginated content, use a loop to generate URLs dynamically:

base_url = "https://example.com/products?page={}"
urls = [base_url.format(i) for i in range(1, 6)]  # Pages 1 to 5

Step 7: Scrape JavaScript-Rendered Content (Optional)

For sites that rely on JavaScript (e.g., React or Angular), use a headless browser like Playwright or Selenium. While not directly asyncio-compatible, Playwright has async support:

from playwright.async_api import async_playwright

async def scrape_js_content():
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto("https://example.com/dynamic-page")
        content = await page.content()
        await browser.close()
        return content

Note: This is beyond asyncio’s scope but complements it. Use this only when necessary, as it adds complexity and resource usage.

Best Practices for Async Web Scraping

✅ Use `async/await` Everywhere

Always use the async/await syntax for asynchronous functions, even if they’re just wrapping simple I/O operations.

✅ Limit Concurrency

Use semaphores or asyncio.Semaphore to control the number of simultaneous requests. Too many can lead to rate limits or server bans.

✅ Handle Timeouts and Errors

Wrap your code in try/except blocks to handle network failures, timeouts, and malformed HTML gracefully.

✅ Avoid Blocking Code

Never use synchronous code inside async functions (e.g., time.sleep()). Instead, use asyncio.sleep().

✅ Monitor Resource Usage

Asynchronous scrapers can consume significant memory if not managed. Use asyncio’s gather with limits or monitor tasks with asyncio.tasks.all_tasks().

Next Steps

Now that you’ve mastered the basics of asyncio for web scraping, consider exploring the following advanced topics:

Distributed Scraping: Use tools like Celery or RabbitMQ to distribute scraping tasks across multiple machines.
Headless Browsers: Learn how to scrape JavaScript-rendered content with Playwright or Selenium.
Data Storage: Store scraped data asynchronously in databases like PostgreSQL, MongoDB, or Redis.
Respectful Scraping: Learn to use robots.txt, rate limits, and IP rotation to scrape ethically and legally.

With these skills, you’ll be well on your way to becoming a master of asynchronous web scraping. Now go build something amazing! 🚀

Built by N3X1S INTELLIGENCE — We build production-grade scrapers. Need data extracted? Hire us on Fiverr.

How to Handle Pagination in Web Scraping (2026 Guide)

Max Klein — Mon, 02 Mar 2026 13:10:07 +0000

How to Handle Pagination in Web Scraping (2026 Guide)

Web scraping is a powerful tool for extracting data from the web, but one of the most common challenges developers face is pagination. Whether you're scraping product listings, blog posts, or user profiles, websites often split content across multiple pages to improve performance and user experience. If you don't handle pagination correctly, you'll miss out on critical data—or worse, trigger anti-scraping measures that block your requests.

In this guide, we’ll walk you through how to handle pagination in web scraping in 2026, covering everything from basic strategies to advanced techniques using Python. By the end, you'll have a robust framework for scraping paginated content efficiently and ethically.

Understanding Pagination in Web Scraping

Pagination refers to the process of dividing content into discrete pages, often using numbered links (e.g., page=1, page=2) or infinite scroll. Here are the common types of pagination you’ll encounter:

1. Numbered Pagination

Example: https://example.com/products?page=1
Easy to handle with loops and URL parameter manipulation.

2. Infinite Scroll

Example: A blog that loads more posts as you scroll down.
Requires JavaScript rendering (e.g., with Selenium or Playwright).

3. API-Based Pagination

Example: A REST API that returns data in chunks using offset or limit parameters.
Often easier to scrape than HTML pages.

4. Dynamic Pagination (e.g., AJAX)

Example: A search result page that loads new results via AJAX when you click "Next."
Can be tricky to detect without inspecting network requests.

Practical Code Examples

Let’s dive into real-world code examples. We’ll cover numbered pagination, infinite scroll, and API-based pagination using Python.

### Example 2: Infinite Scroll with `Playwright`

Infinite scroll is common on social media and e-commerce sites. Here’s how to handle it using Playwright:

from playwright.sync_api import sync_playwright
import time

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()

    page.goto("https://example.com/infinite-scroll")

    # Scroll until all content is loaded
    last_height = page.evaluate("document.body.scrollHeight")
    while True:
        page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
        time.sleep(2)  # Wait for new content to load
        new_height = page.evaluate("document.body.scrollHeight")

        if new_height == last_height:
            break
        last_height = new_height

    # Extract all items (example selector)
    items = page.query_selector_all(".item")
    for item in items:
        print(item.text_content())

    browser.close()

Tip: Use headless=False for debugging. In production, set headless=True for faster execution.

Tips, Warnings, and Best Practices

### 1. Respect Website Policies

Always check robots.txt and terms of service.
Avoid scraping sensitive data (e.g., personal information).

### 2. Use Headers and User Agents

Mimic a real browser to avoid being blocked. Example:

  headers = {
      "User-Agent": "Mozilla/5.0",
      "Accept-Language": "en-US,en;q=0.9",
  }

### 3. Implement Delays

Add time.sleep() between requests to avoid overwhelming servers.
A delay of 2–5 seconds is generally safe.

### 4. Handle Dynamic Content

For JavaScript-rendered pages, use Selenium or Playwright.
Avoid using requests alone for dynamic content.

### 5. Use Proxies and Rotate IPs

If you’re scraping a large number of pages, use a proxy service to avoid IP bans.
Libraries like httpx and fake_useragent can help.

### 6. Error Handling

Always include try-except blocks for robustness:

  try:
      response = requests.get(url)
      response.raise_for_status()
  except requests.exceptions.RequestException as e:
      print(f"Request failed: {e}")

Next Steps

Now that you’ve mastered pagination, consider exploring these advanced topics:

Scrapy Framework: Learn how to use Scrapy’s built-in pagination with Rule and LinkExtractor.
Headless Browser Optimization: Improve performance with Playwright or Selenium configurations.
CAPTCHA Bypassing: Explore tools like 2Captcha or Anti-Captcha APIs.
Distributed Scraping: Use Scrapy-Redis or Apache Nutch to scale your scrapers.
Data Storage: Learn to store scraped data in databases like PostgreSQL or MongoDB.

Remember: Web scraping is a powerful tool, but it must be used responsibly. Always prioritize legal compliance and ethical considerations.

Happy scraping! 🕵️‍♂️

Built by N3X1S INTELLIGENCE — We build production-grade scrapers. Need data extracted? Hire us on Fiverr.

Building a Real Estate Price Tracker with Python

Max Klein — Mon, 02 Mar 2026 12:50:21 +0000

In today's fast-paced real estate market, staying ahead of price trends can be the difference between a smart investment and a costly mistake. This tutorial guides you through building a real estate price tracker using Python, combining web scraping, data analysis, and visualization.

Prerequisites

pip install requests beautifulsoup4 pandas matplotlib seaborn schedule plotly

Step 1: Scraping Real Estate Listings

import requests
from bs4 import BeautifulSoup
import pandas as pd

def scrape_real_estate_data(url):
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
    }
    response = requests.get(url, headers=headers)
    if response.status_code != 200:
        return pd.DataFrame()

    soup = BeautifulSoup(response.text, "html.parser")
    properties = []
    for item in soup.select(".property-listing .item"):
        title = item.select_one(".title").text.strip()
        price = item.select_one(".price").text.strip()
        properties.append({"Title": title, "Price": price})

    return pd.DataFrame(properties)

Tips: Use headers to mimic browser requests. Respect rate limits. Test selectors with browser dev tools.

Step 2: Data Cleaning

def clean_real_estate_data(df):
    df.dropna(subset=["Price"], inplace=True)
    df["Price"] = df["Price"].str.replace("[^0-9]", "", regex=True)
    df["Price"] = pd.to_numeric(df["Price"], errors="coerce")
    df.dropna(subset=["Price"], inplace=True)
    return df

Step 3: Visualization

import plotly.express as px

fig = px.scatter(cleaned_data, x="Location", y="Price", 
                 size="Price", color="Price",
                 hover_name="Title", 
                 title="Real Estate Prices by Location")
fig.show()

Step 4: Automation

import schedule
import time

def job():
    print("Updating real estate data...")
    data = scrape_real_estate_data(url)
    cleaned = clean_real_estate_data(data)
    # Save, analyze, alert...

schedule.every().day.at("08:00").do(job)

while True:
    schedule.run_pending()
    time.sleep(1)

Step 5: Deploy as a Web App

from flask import Flask, render_template
import pandas as pd

app = Flask(__name__)

@app.route("/")
def index():
    df = pd.read_csv("real_estate_data.csv")
    return render_template("index.html", data=df.to_dict())

if __name__ == "__main__":
    app.run(debug=True)

Conclusion

You've built a real estate price tracker that scrapes, cleans, analyzes, and visualizes data. Extend it with ML predictions, database storage, or mobile deployment.

Need professional web scraping for real estate data or any other industry? Check out N3X1S INTELLIGENCE on Fiverr — same-day delivery, anti-bot bypass, clean data guaranteed.

How to Store Scraped Data: CSV vs JSON vs Database

Max Klein — Mon, 02 Mar 2026 12:50:03 +0000

Web scraping is a powerful tool for extracting valuable information from the web, but the real challenge lies in what happens after the data is collected. How you store scraped data determines how easy it is to analyze, query, and reuse that information. In this tutorial, we'll walk through three popular storage options—CSV, JSON, and databases—and help you choose the best approach for your project.

Why Storage Matters

Storing scraped data is more than just saving files to your computer. The right storage method can:

Improve data integrity and reusability
Enable efficient querying and scalability
Reduce duplicate data and processing overhead

Let's break down the three options and see how they stack up.

Saving Scraped Data to CSV

CSV is the simplest format. Here's how to save scraped book data:

import csv

books = [
    {"title": "Python Crash Course", "author": "Eric Matthes", "price": "$29.99"},
    {"title": "Automate the Boring Stuff", "author": "Al Sweigart", "price": "$19.99"},
    {"title": "Clean Code", "author": "Robert C. Martin", "price": "$39.99"},
]

with open("books.csv", "w", newline="", encoding="utf-8") as csvfile:
    fieldnames = ["title", "author", "price"]
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    for book in books:
        writer.writerow(book)

Tip: Always use newline="" when writing CSV files in Python to avoid extra blank lines.

When to use CSV: Simple, flat data with consistent columns. Quick exports for spreadsheets.

Saving Scraped Data to JSON

JSON handles nested structures much better:

import json

books = [
    {
        "title": "Python Crash Course",
        "author": "Eric Matthes",
        "price": "$29.99",
        "categories": ["Programming", "Education"]
    },
    {
        "title": "Automate the Boring Stuff",
        "author": "Al Sweigart",
        "price": "$19.99",
        "categories": ["Automation", "Scripting"]
    }
]

with open("books.json", "w", encoding="utf-8") as jsonfile:
    json.dump(books, jsonfile, indent=4)

When to use JSON: Nested or hierarchical data, API responses, config files.

Saving Scraped Data to a Database

For large-scale projects, databases are the way to go. Here's SQLite:

import sqlite3

conn = sqlite3.connect("books.db")
cursor = conn.cursor()

cursor.execute("""
CREATE TABLE IF NOT EXISTS books (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    title TEXT,
    author TEXT,
    price TEXT
)
""")

books = [
    ("Python Crash Course", "Eric Matthes", "$29.99"),
    ("Automate the Boring Stuff", "Al Sweigart", "$19.99"),
]

cursor.executemany("INSERT INTO books (title, author, price) VALUES (?, ?, ?)", books)
conn.commit()
conn.close()

When to use databases: Large datasets, need for querying, deduplication, multi-user access.

Quick Comparison

CSV: Simple, universal, but no nesting, no querying
JSON: Flexible, nested data, but large files are slow
Database: Scalable, queryable, but more setup overhead

Conclusion

CSV, JSON, and databases each have their place. CSV for quick exports, JSON for structured data, databases for production workloads. Choose based on your project's scale and complexity.

Need professional web scraping with clean, structured data delivery? Check out N3X1S INTELLIGENCE on Fiverr — we handle scraping, cleaning, and delivery in any format you need.

How to Schedule Python Scripts to Run Automatically

Max Klein — Mon, 02 Mar 2026 12:30:28 +0000

Automating repetitive tasks is one of the most powerful ways to boost productivity. In this tutorial, we'll walk through five proven methods to schedule Python scripts.

Method 1: Using Cron (Unix/macOS)

Create Your Script

# automate.py
from datetime import datetime

def log_time():
    print(f"[{datetime.now()}] Task executed")

if __name__ == "__main__":
    log_time()

Edit the Crontab

crontab -e

Add this line to run every minute:

* * * * * /usr/bin/python3 /path/to/automate.py >> /path/to/logfile.txt 2>&1

Warning: Cron does not support sub-second intervals.

Method 2: Task Scheduler (Windows)

Create a .bat file:

@echo off
python C:\path\to\automate.py

Open Task Scheduler > Create Basic Task > Set trigger > Start a Program > Browse to .bat file

Method 3: APScheduler

pip install apscheduler

from apscheduler.schedulers.background import BackgroundScheduler
from datetime import datetime
import time

def job():
    print(f"[{datetime.now()}] Job executed")

scheduler = BackgroundScheduler()
scheduler.add_job(job, 'interval', seconds=10)
scheduler.start()

try:
    while True:
        time.sleep(1)
except KeyboardInterrupt:
    scheduler.shutdown()

Method 4: The schedule Library

pip install schedule

import schedule
import time
from datetime import datetime

def job():
    print(f"[{datetime.now()}] Job executed")

schedule.every(5).seconds.do(job)

while True:
    schedule.run_pending()
    time.sleep(1)

Best Practices

1. Use Logging Instead of print()

import logging
logging.basicConfig(filename='app.log', level=logging.INFO)

def job():
    logging.info("Job executed at %s", datetime.now())

2. Handle Errors Gracefully

def job():
    try:
        pass  # Your code here
    except Exception as e:
        logging.error("Error: %s", e)

3. Use Absolute Paths

Always use absolute paths for Python, the script file, and dependencies.

4. Test Before Scheduling

Run your script manually first.

Next Steps

Persisting Jobs: Use APScheduler's SQLAlchemyJobStore
Distributed Scheduling: Use Celery with RabbitMQ/Redis
Cloud Automation: Deploy on AWS, GCP, or Azure

Need automated data pipelines or scheduled scrapers? N3X1S INTELLIGENCE builds production-ready automation. Hire us on Fiverr →

Data Enrichment: How to Add Value to Raw Scraped Data

Max Klein — Mon, 02 Mar 2026 12:29:57 +0000

Imagine you've spent hours scraping data from a website—product names, prices, and descriptions. But when you look at the dataset, it feels incomplete. This is the reality of raw scraped data: useful, but limited. To unlock its full potential, you need data enrichment.

In this tutorial, we'll walk through data enrichment techniques using Python.

What Is Data Enrichment?

Data enrichment is the process of enhancing raw data by adding relevant information from external sources:

Geocoding addresses to get latitude/longitude
Appending product categories using a database
Validating email formats with regex or external tools
Enhancing user profiles with demographic data from APIs

Why Enrich Data?

Improves decision-making: Enriched data provides context
Reduces errors: Cleaning and validating data early prevents downstream issues
Boosts ML models: Feature-rich datasets yield better predictions
Unlocks new insights: Merging datasets reveals hidden patterns

Practical Example: Enriching Scraped Product Data

Assume you've scraped products from an e-commerce site:

Step 1: Install Required Libraries

pip install pandas requests

Step 2: Load Raw Data

import pandas as pd

df = pd.read_csv("products.csv")
print(df.head())

Step 3: Enrich with Product Categories

import requests

def get_category(product_name):
    url = "https://api.example.com/product-category"
    payload = {"product_name": product_name}
    response = requests.post(url, json=payload)
    if response.status_code == 200:
        return response.json().get("category", "Unknown")
    return "Unknown"

df["Category"] = df["Product Name"].apply(get_category)

Step 4: Add Brand Names

def get_brand(product_name):
    url = "https://api.example.com/product-brand"
    payload = {"product_name": product_name}
    response = requests.post(url, json=payload)
    if response.status_code == 200:
        return response.json().get("brand", "Unknown")
    return "Unknown"

df["Brand"] = df["Product Name"].apply(get_brand)

Step 5: Save the Enriched Data

df.to_csv("enriched_products.csv", index=False)

Merging Datasets with Pandas

customers = pd.read_csv("customers.csv")
demographics = pd.read_csv("demographics.csv")

merged_data = pd.merge(customers, demographics, on="email", how="left")
print(merged_data.head())

Best Practice: Always check for missing values after merging with merged_data.isnull().sum().

Common Pitfalls

Overloading Your Dataset — Prioritize the most impactful enrichments first
Ignoring Data Quality — Always validate and clean API results
Not Documenting Your Process — Keep track for reproducibility

Next Steps

Automate enrichment with Apache Airflow or Prefect
Explore APIs like OpenWeather, Yelp, or LinkedIn
Learn ETL processes for large-scale workflows
Use pydantic or jsonschema for data validation

Need professional data scraping and enrichment? N3X1S INTELLIGENCE delivers clean, enriched datasets from any source. Hire us on Fiverr →

How to Scrape Yelp with Python: Step-by-Step Guide 2026

Max Klein — Mon, 02 Mar 2026 12:10:08 +0000

How to Scrape Yelp with Python (Step-by-Step Guide 2026)

In 2026, the world of web scraping has evolved dramatically. With more businesses relying on platforms like Yelp for reviews, listings, and customer insights, the ability to extract this data programmatically has become a valuable skill. Whether you're a developer building a competitive analysis tool, a data scientist gathering market trends, or a small business owner optimizing your Yelp presence, scraping Yelp with Python can unlock a treasure trove of information. However, modern websites like Yelp are increasingly complex, using JavaScript to dynamically load content and sophisticated anti-scraping measures. This tutorial will guide you through the process of scraping Yelp with Python in 2026, covering everything from setup to advanced techniques.

Step 1: Setting Up Your Python Environment

Start by creating a new Python file (e.g., yelp_scraper.py) and importing the necessary libraries:

import requests
from bs4 import BeautifulSoup
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
import time

Warning: Selenium requires a compatible browser driver (e.g., ChromeDriver). Ensure your browser and driver versions match, and keep them updated.

Step 3: Parsing HTML with BeautifulSoup

Once you have the HTML content, use BeautifulSoup to extract data. Here's how to extract restaurant names, ratings, and addresses:

soup = BeautifulSoup(html_content, "lxml")

businesses = soup.find_all("div", class_="business-identifier")

data = []
for business in businesses:
    name = business.find("h2", class_="business-name").text.strip()
    rating = business.find("div", class_="i-stars")
    if rating:
        rating = rating["aria-label"].split(" ")[0]
    else:
        rating = "N/A"
    address = business.find("span", class_="address").text.strip()

    data.append({
        "Name": name,
        "Rating": rating,
        "Address": address
    })

# Convert to a pandas DataFrame
df = pd.DataFrame(data)
print(df.head())

Best Practice: Use lxml as the parser for faster performance. If lxml is unavailable, html.parser is a fallback option.

Step 5: Scraping Multiple Pages and Pagination

Yelp search results are paginated. To scrape all pages, extract the "Next" button's URL and loop through each page:

base_url = "https://www.yelp.com/search?find_desc=Restaurants&find_loc=San+Francisco"
page_number = 1

all_data = []

while True:
    url = f"{base_url}&start={page_number * 10}"
    response = requests.get(url, headers=headers)

    if response.status_code != 200:
        break  # Stop if the page is not found

    soup = BeautifulSoup(response.text, "lxml")
    businesses = soup.find_all("div", class_="business-identifier")

    for business in businesses:
        # Extract data as before...
        pass

    # Check for "Next" button
    next_button = soup.find("a", class_="next")
    if not next_button:
        break

    page_number += 1

# Save all_data to a DataFrame or file

Best Practice: Add a time.sleep(2) between requests to avoid being rate-limited. Use proxies or user-agent rotation for large-scale scraping.

Step 7: Advanced Techniques (Optional)

Using Proxies

To avoid IP blocking, use proxy services like BrightData or Luminati. Example with requests:

proxies = {
    "http": "http://username:password@proxy_ip:port",
    "https": "http://username:password@proxy_ip:port"
}
response = requests.get(url, headers=headers, proxies=proxies)

Rotating User-Agents

Use a list of user-agents and rotate them for each request:

import random

user_agents = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    # Add more user-agents...
]

headers = {
    "User-Agent": random.choice(user_agents)
}

Next Steps

Now that you've mastered the basics, here are some ideas for further exploration:

Automate Data Collection: Build a script that runs daily to monitor Yelp for new reviews or competitors.
Integrate with a Database: Store scraped data in a PostgreSQL or MongoDB database for advanced analysis.
Use Scrapy: Explore the Scrapy framework for large-scale web scraping projects.
Learn API Usage: Switch to Yelp's official API for legal and efficient data access.
Explore Rate Limiting: Implement advanced techniques like token bucket algorithms to manage request rates.

By continuing to refine your skills, you'll be well-equipped to tackle any web scraping challenge in 2026 and beyond. Happy coding! 🚀

🚀 Need professional web scraping? N3X1S INTELLIGENCE on Fiverr — fast, reliable data extraction from any website.

Building a Price Tracker with Python and BeautifulSoup

Max Klein — Mon, 02 Mar 2026 12:09:57 +0000

Building a Price Tracker with Python and BeautifulSoup

In today’s fast-paced digital marketplace, staying ahead of price fluctuations is a game-changer. Whether you're a savvy shopper, an e-commerce entrepreneur, or a data analyst, the ability to track product prices in real time can save you money, optimize inventory, and uncover market trends. But manually checking prices across dozens of websites is tedious, error-prone, and time-consuming. What if you could automate this process with a few lines of code? That’s where Python and BeautifulSoup come into play. In this tutorial, we’ll walk you through building a price tracker that scrapes product prices from e-commerce sites, stores them securely, and even visualizes trends over time. By the end, you’ll have a powerful tool that works seamlessly in the background, freeing you up to focus on what matters most.

Prerequisites

Before diving into the code, ensure your environment is set up correctly. Here’s what you’ll need:

Software Requirements

Python 3.x installed on your machine (preferably 3.7 or newer).
A code editor or IDE (e.g., VS Code, PyCharm, or Jupyter Notebook).
A modern web browser (for inspecting HTML elements during scraping).

Python Libraries

Install the following Python packages using pip:

pip install beautifulsoup4 requests pandas matplotlib

BeautifulSoup: For parsing HTML and XML documents.
Requests: For sending HTTP requests to fetch web pages.
Pandas: For organizing and storing scraped data.
Matplotlib: For visualizing price trends (optional but recommended).

Web Scraping Ethics

Respect website terms of service and robots.txt files.
Avoid overloading servers with excessive requests (use rate limiting).
Use headers to mimic a real browser (see tips below).

Setting Up Your Project

Create a new directory for your project and initialize it with a requirements.txt file:

mkdir price-tracker
cd price-tracker
touch requirements.txt

Add the required libraries to requirements.txt:

beautifulsoup4
requests
pandas
matplotlib

Now, create a Python file (e.g., price_tracker.py) and import the necessary modules:

import requests
from bs4 import BeautifulSoup
import pandas as pd
import matplotlib.pyplot as plt
import time

This setup will form the backbone of your price tracker.

Step 1: Scraping Product Prices with BeautifulSoup

Let’s start by scraping a product page. We’ll use a fictional e-commerce site for this example, but the same principles apply to real websites like Amazon, eBay, or any HTML-based store.

Example: Scraping a Product Page

def fetch_product_price(url):
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
    }
    try:
        response = requests.get(url, headers=headers, timeout=10)
        response.raise_for_status()
    except requests.RequestException as e:
        print(f"Error fetching {url}: {e}")
        return None

    soup = BeautifulSoup(response.text, "html.parser")

    # Example: Extract product name and price
    product_name = soup.find("h1", class_="product-title").text.strip()
    price_element = soup.find("span", class_="price")
    if price_element:
        price = price_element.text.strip()
        # Clean price by removing non-numeric characters
        cleaned_price = float(price.replace("$", "").replace(",", ""))
        return {"name": product_name, "price": cleaned_price}
    else:
        print("Price not found on the page.")
        return None

Key Notes

User-Agent Headers: Websites often block scrapers by detecting bot traffic. Use a browser’s User-Agent string to mimic a real user.
Error Handling: Always handle exceptions for network errors, timeouts, or malformed HTML.
CSS Selectors: Inspect the HTML of the target site (using browser dev tools) to identify the correct CSS classes or IDs for price and product name.

Step 2: Handling Dynamic Content (Advanced)

While BeautifulSoup is great for static HTML, many modern websites load content dynamically using JavaScript (e.g., React, Vue, or Angular). In such cases, BeautifulSoup alone won’t work. Here’s how to handle it:

Option 1: Use Selenium (For JavaScript-Heavy Sites)

Install Selenium:

pip install selenium

Then, use it to render JavaScript:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

def fetch_dynamic_price(url):
    driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
    driver.get(url)
    time.sleep(5)  # Wait for JavaScript to load
    price = driver.find_element("css selector", ".price").text
    driver.quit()
    return price

🚨 Warning: Selenium is slower and resource-heavy. Use it only when necessary.

Option 2: Use API Endpoints (If Available)

Some sites expose price data via REST APIs. Check the site’s network traffic in dev tools for API endpoints and use requests to fetch JSON data directly.

Step 3: Storing Data in a Database

Storing scraped data ensures you can track prices over time. We’ll use SQLite for simplicity, but you could also use PostgreSQL, MySQL, or even a CSV file.

Example: Saving to SQLite

import sqlite3

def save_to_database(product_data):
    conn = sqlite3.connect("prices.db")
    c = conn.cursor()

    # Create table if it doesn't exist
    c.execute('''
        CREATE TABLE IF NOT EXISTS product_prices (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            product_name TEXT,
            price REAL,
            timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
        )
    ''')

    # Insert data
    c.execute('''
        INSERT INTO product_prices (product_name, price)
        VALUES (?, ?)
    ''', (product_data["name"], product_data["price"]))

    conn.commit()
    conn.close()

Alternative: Using Pandas

You can also store data in CSV or JSON format:

def save_to_csv(product_data):
    df = pd.DataFrame([product_data])
    df.to_csv("product_prices.csv", mode="a", header=not pd.io.common.file_exists("product_prices.csv"), index=False)

Step 4: Automating the Price Tracker

To keep your tracker running continuously, use a scheduler or cron job.

Example: Using APScheduler

Install APScheduler:

pip install apscheduler

Then, set up a recurring job:

from apscheduler.schedulers.blocking import BlockingScheduler

def job():
    product_url = "https://example.com/product/123"
    product_data = fetch_product_price(product_url)
    if product_data:
        save_to_database(product_data)
        print("Data saved successfully.")

if __name__ == "__main__":
    scheduler = BlockingScheduler()
    scheduler.add_job(job, 'interval', hours=1)  # Run every hour
    try:
        scheduler.start()
    except KeyboardInterrupt:
        print("Scheduler stopped.")

Tips

Rate Limiting: Add a time.sleep() delay between requests to avoid being blocked.
Logging: Use Python’s logging module to track errors and success messages.
Configuration: Store URLs and database settings in a config file or environment variables.

Step 5: Visualizing Price Trends

Now that we’re collecting data, let’s visualize it. We’ll use Matplotlib to plot price trends over time.

Example: Plotting Historical Prices

def plot_price_history(product_name):
    conn = sqlite3.connect("prices.db")
    df = pd.read_sql_query("SELECT * FROM product_prices WHERE product_name = ?", conn, params=(product_name,))
    conn.close()

    if not df.empty:
        plt.figure(figsize=(10, 5))
        plt.plot(df["timestamp"], df["price"], marker="o")
        plt.title(f"Price Trend for {product_name}")
        plt.xlabel("Date")
        plt.ylabel("Price ($)")
        plt.grid(True)
        plt.xticks(rotation=45)
        plt.tight_layout()
        plt.show()
    else:
        print(f"No data found for {product_name}.")

Enhancements

Use Pandas to aggregate data by day or week.
Add interactivity with Plotly for web-based dashboards.
Export charts as PNG or PDF for reports.

Conclusion

You’ve now built a fully functional price tracker using Python and BeautifulSoup! This tool can monitor product prices in real time, store historical data, and even visualize trends. Whether you’re tracking competitors, optimizing your pricing strategy, or just saving money on your next purchase, this script is a powerful ally.

Next Steps

Ready to take your price tracker to the next level? Here are some ideas:

Add Email Alerts: Use SMTP or services like SendGrid to notify you when prices drop.
Deploy as a Web App: Use Flask or Django to create a dashboard for viewing price trends.
Scrape Multiple Products: Loop through a list of URLs and store data in a structured format.
Integrate with APIs: Use Stripe or Shopify APIs to sync data with your e-commerce platform.
Use Cloud Storage: Store data in AWS S3, Google Cloud, or MongoDB for scalability.

As you explore these enhancements, always remember to respect website policies and avoid unethical scraping practices. Happy coding!

🚀 Need professional web scraping? N3X1S INTELLIGENCE on Fiverr — fast, reliable data extraction from any website.

DEV Community: Max Klein

How spreading activation beats keyword search for LLM agents

The idea

How it works

The 13 tools

Real example

Why not vector search?

Try it

Building a Stock Price Bot with Python: Complete Tutorial

Prerequisites

📦 Software and Tools

📌 Required Libraries

Setting Up Your Development Environment

Step 1: Fetching Real-Time Stock Data with yfinance

✅ Example: Get the Current Price of a Stock

🔍 Explanation

Step 2: Monitoring Price Changes and Sending Alerts

🚨 Example: Price Alert System

⚠️ Important Notes

Step 3: Logging Historical Data for Analysis

📊 Example: Save Historical Data

📈 Enhancements

Step 4: Automating the Bot with a Scheduler

⏰ Example: Schedule Daily Price Checks

Step 5: Deploying Your Bot

🚀 Deployment Options

🛡️ Best Practices for Deployment

Conclusion

Next Steps

Final Thoughts

Rotating Proxies for Web Scraping: Complete Guide

Prerequisites

What Are Rotating Proxies?

Definition and Purpose

Use Cases for Rotating Proxies

How Rotating Proxies Work

The Mechanics Behind the Scenes

Key Components of a Rotating Proxy System

Benefits of Using Rotating Proxies

1. Avoid IP Bans and Account Lockouts

2. Bypass Rate Limits

3. Improve Scraping Efficiency

4. Enhance Data Accuracy

Choosing the Right Rotating Proxy Service

Factors to Consider

Popular Rotating Proxy Providers

Practical Code Examples: Using Rotating Proxies in Python

Example 1: Basic Rotating Proxy with requests

Example 2: Rotating Proxies with a Proxy Service API

Best Practices for Using Rotating Proxies

1. Rotate Proxies at Appropriate Intervals

2. Use User-Agent Rotation

3. Respect Website Policies

4. Monitor and Handle Errors

Advanced Techniques: Combining Proxies with Web Scraping Frameworks

Using Scrapy with Rotating Proxies

Conclusion

Next Steps

How to Scrape Google Search Results Legally

Prerequisites

Tools and Knowledge Required

Software Installation

Legal Considerations: Why You Can’t Just Scrape Google

Method 1: Using Google’s Custom Search API (Recommended)

Step 1: Set Up a Google Cloud Project

Step 2: Use the API in Python

Method 2: Scraping Google Search Results with Selenium (For Advanced Users)

Step 1: Set Up Selenium

Step 2: Automate Google Search

Best Practices for Legal and Ethical Scraping

1. Respect Robots.txt

2. Implement Rate Limiting

3. Rotate User-Agents

4. Handle CAPTCHA and Blocks Gracefully

Advanced Techniques: Combining APIs and Scraping

Conclusion

Next Steps

Python asyncio for Web Scraping: Speed Up 10x

Python asyncio for Web Scraping: Speed Up 10x

Understanding AsyncIO and Its Role in Web Scraping

Example 1: Basic Rotating Proxy with `requests`

Using `Scrapy` with Rotating Proxies

✅ Use `async/await` Everywhere

### Example 2: Infinite Scroll with `Playwright`