In today's highly competitive e-commerce market, keeping track of real-time pricing has become a key factor for sellers to stay competitive. As one of the world's largest C2C and B2C e-commerce marketplaces, eBay processes hundreds of millions of product transactions every day. For operations teams, manually refreshing pages to monitor competitor pricing is not only inefficient but also increases the risk of missing optimal repricing opportunities.
This article provides a step-by-step guide to building a complete eBay price monitoring tool with Python. It covers the entire eBay data scraping workflow—from environment setup, proxy configuration, and data extraction to automated storage and email notifications. It also addresses the most common access restrictions and blocking issues encountered during data collection, helping you build a stable and cost-effective eBay price monitoring system.
I. Why Is eBay Data Scraping Valuable?
For e-commerce sellers, operations teams, and data analysts, the value of eBay data scraping goes far beyond simply checking prices. It enables you to:
● Capture competitor price adjustments as soon as they occur and avoid losing sales due to delayed pricing updates.
● Collect sales volume, review counts, and other metrics in bulk to identify trending categories and potential winning products.
● Replace manual data entry by automatically extracting structured data such as titles, images, and specifications.
● Build historical price trend models that support dynamic pricing strategies.
Based on these use cases, the goal of this article is to build a lightweight eBay price monitoring tool using Python and use proxies to overcome common access restrictions, making eBay data scraping more stable and efficient.
II. eBay Price Monitoring: Scraping Product Data with Python
Next, we'll build the entire system step by step. Each section includes runnable Python code that can be executed in sequence.
1. Prepare the Tools
Before starting eBay data scraping, install the following dependencies:
pip install requests beautifulsoup4 lxml schedule smtplib
requests is used for sending HTTP requests, beautifulsoup4 and lxml are used for parsing HTML, schedule handles task scheduling, and Python's built-in smtplib module is used for sending email alerts without requiring additional installation.
2. Retrieve eBay Product Pages
eBay uses User-Agent detection to identify bots. Using default request headers often results in 403 errors or CAPTCHA challenges. The following code uses browser-like request headers to retrieve page content:
import requests
from bs4 import BeautifulSoup
from requests.adapters import HTTPAdapter
from urllib3.util import Retry
def get_ebay_page(url, proxy=None):
# 1. Initialize Session (automatically manage cookies)
session = requests.Session()
# 2. Configure retry mechanism (prevent temporary network interruptions)
retries = Retry(total=3, backoff_factor=1, status_forcelist=[500, 502, 503, 504])
session.mount('http://', HTTPAdapter(max_retries=retries))
session.mount('https://', HTTPAdapter(max_retries=retries))
# 3. More realistic browser headers
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
"Cache-Control": "max-age=0",
"Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1"
}
session.headers.update(headers)
# 4. Normalize proxy format
proxies = None
if proxy:
# Automatically add protocol if missing
if not proxy.startswith(('http://', 'https://')):
proxy = f"http://{proxy}"
proxies = {"http": proxy, "https": proxy}
try:
# 5. Optional: Visit the homepage first to obtain initial cookies (important for anti-bot protection)
# session.get("https://www.ebay.com", proxies=proxies, timeout=5)
# 6. Request the product page
response = session.get(url, proxies=proxies, timeout=10)
response.raise_for_status()
# eBay-specific validation: status code may be 200 while the page contains a security check
if "captcha" in response.url or "Robot Check" in response.text:
print("[Warning] Blocked by eBay Bot Check. Please switch proxy IPs or refresh cookies.")
return None
return response.text
except requests.exceptions.RequestException as e:
print(f"[Error] Failed to request page: {e}")
return None
# ----- Test Run -----
if __name__ == "__main__":
test_url = "https://www.ebay.com/itm/225643445557" # Replace with a real eBay product URL
html_content = get_ebay_page(test_url)
if html_content:
soup = BeautifulSoup(html_content, "html.parser")
# Print page title to verify successful retrieval
print("Page Title:", soup.title.text.strip() if soup.title else "Title Not Found")
Keep in mind that eBay page structures may vary depending on product categories. Before scraping, inspect the target page using browser developer tools and verify the CSS selectors of the elements you need.
3. Configure Proxies
One of the most common obstacles in eBay data scraping is IP blocking caused by frequent requests. Using rotating residential proxies allows requests to be distributed across multiple IP addresses, reducing the risk of interruptions caused by IP bans. This approach is particularly suitable for long-term eBay price monitoring projects.
For example, IPFoxy Proxies offers rotating residential proxies that can serve as a professional solution for large-scale data collection.
Obtain Proxy Information
Through IPFoxy Proxies, acquire a rotating residential proxy and configure parameters such as state/city location, protocol type, session rotation mode, and proxy format. After configuration, you will receive proxy connection credentials.
Configure Proxies in Python
Paste the proxy connection information obtained from IPFoxy Proxies into the following example:
import urllib.request
if __name__ == '__main__':
proxy = urllib.request.ProxyHandler({
'https': 'username:password@gate-us-ipfoxy.io:58688',
'http': 'username:password@gate-us-ipfoxy.io:58688',
})
opener = urllib.request.build_opener(proxy, urllib.request.HTTPHandler)
urllib.request.install_opener(opener)
content = urllib.request.urlopen('http://www.ip-api.com/json').read()
print(content)
After running the code, you should see that the outbound IP address has changed. This indicates that the proxy configuration is working correctly and you can proceed to the next step.
4. Extract Product Information
Once the page is successfully retrieved, use BeautifulSoup to parse the HTML and extract key fields such as product title, price, and stock availability:
import re
from bs4 import BeautifulSoup
def extract_product_info(html):
# Recommended parser: html.parser (or lxml if installed)
soup = BeautifulSoup(html, "html.parser")
# 1. Product title: extract directly or from nested span
title_element = soup.find("h1", class_="x-item-title__mainTitle")
if title_element:
# eBay often places the title inside a span with class ux-textspans
sub_span = title_element.find("span", class_="ux-textspans")
title_text = sub_span.get_text(strip=True) if sub_span else title_element.get_text(strip=True)
else:
title_text = "N/A"
# 2. Current price: extract and clean numeric value for monitoring
price_element = soup.find("div", class_="x-price-primary")
price_text = "N/A"
raw_price = "N/A"
if price_element:
raw_price = price_element.get_text(strip=True) # Example: "US $24.99" or "C $35.00/ea"
# Regular expression: extract numeric price value
price_match = re.search(r'\d+(?:\.\d+)?', raw_price.replace(',', ''))
if price_match:
# Converted to numeric string "24.99"
price_text = price_match.group(0)
# 3. Stock status: use fallback selectors for better reliability
stock_text = "Available" # Assume available by default
# Try the original class name first
stock_element = soup.find("div", class_="d-quantity__availability")
# Fallback option: eBay may use alternative wrappers
if not stock_element:
stock_element = soup.find(class_=re.compile(r".*quantity__availability.*"))
if stock_element:
stock_text = stock_element.get_text(strip=True)
else:
# Check whether the page contains stock-related keywords
page_text = soup.get_text()
if "Out of stock" in page_text or "This item is out of stock" in page_text:
stock_text = "Out of Stock"
return {
"title": title_text,
"raw_price": raw_price, # Original price text, e.g. "US $24.99"
"clean_price": price_text, # Numeric value, e.g. "24.99"
"stock": stock_text,
}
5. Save Price Data
Store each collected price record in a CSV file for future trend analysis or import into Excel and databases:
import csv
import os
from datetime import datetime
def save_to_csv(data: dict, product_url="N/A", filename="ebay_prices.csv"):
# 1. Define standard CSV fields
fieldnames = ["timestamp", "title", "price", "stock", "url"]
# 2. Safely check whether the file exists
file_exists = os.path.exists(filename)
# 3. Build the row data to be written
row_to_write = {
"timestamp": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
# Use .get(key, default) for fault tolerance
"title": data.get("title", "N/A"),
# Support both clean_price and price fields
"price": data.get("clean_price") or data.get("price") or "N/A",
"stock": data.get("stock", "N/A"),
# Explicitly pass URL or retrieve from data
"url": data.get("url") or product_url
}
# 4. Write data to file
try:
with open(filename, "a", newline="", encoding="utf-8-sig") as f:
# utf-8-sig prevents garbled characters when opening in Excel on Windows
writer = csv.DictWriter(f, fieldnames=fieldnames)
# Write header if the file is newly created
if not file_exists:
writer.writeheader()
# Write row data
writer.writerow(row_to_write)
print(f"[Success] Data appended to {filename}")
except Exception as e:
print(f"[Error] Failed to write CSV file: {e}")
# ----- Demo Test -----
if __name__ == "__main__":
mock_parsed_data = {
"title": "iPhone 15 Pro Max 256GB - Unlocked",
"clean_price": "999.99",
"raw_price": "US $999.99",
"stock": "Limited quantity available"
}
target_url = "https://www.ebay.com/itm/123456789"
# Test save
save_to_csv(mock_parsed_data, product_url=target_url)
Using the timestamp field together with the price field allows you to reconstruct a complete historical pricing timeline and provides a reliable foundation for eBay price monitoring.
6. Monitor Price Changes
Based on the saved historical data, add threshold-based price monitoring logic. When the product price falls below a predefined target value, the system automatically triggers an alert:
import os
import re
import csv
from datetime import datetime
# ==================== Supporting Functions ====================
# (Shortened here for context. Use the optimized versions from previous sections.)
def get_ebay_page(url, proxy=None):
import requests
headers = {"User-Agent": "Mozilla/5.0"}
proxies = {"http": proxy, "https": proxy} if proxy else None
res = requests.get(url, headers=headers, proxies=proxies, timeout=10)
return res.text
def extract_product_info(html):
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
title_el = soup.find("h1", class_="x-item-title__mainTitle")
price_el = soup.find("div", class_="x-price-primary")
title = title_el.get_text(strip=True) if title_el else "Unknown Item"
raw_price = price_el.get_text(strip=True) if price_el else "N/A"
# Extract numeric value using regex
price_match = re.search(r'\d+(?:\.\d+)?', raw_price.replace(',', ''))
clean_price = price_match.group(0) if price_match else "N/A"
return {
"title": title,
"raw_price": raw_price,
"clean_price": clean_price,
"stock": "Available"
}
def save_to_csv(data: dict, product_url, filename):
file_exists = os.path.exists(filename)
with open(filename, "a", newline="", encoding="utf-8-sig") as f:
writer = csv.DictWriter(
f,
fieldnames=["timestamp", "title", "price", "stock", "url"]
)
if not file_exists:
writer.writeheader()
writer.writerow({
"timestamp": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
"title": data["title"],
"price": data["clean_price"],
"stock": data["stock"],
"url": product_url
})
# ==================== Improved Monitoring and Alert Logic ====================
def send_alert(title, current_price, url):
"""
Trigger alert workflow (can be extended to email, Slack, Teams, etc.)
"""
print("\n" + "=" * 40)
print(" [Price Alert Triggered] Low-price item detected!")
print(f" Product: {title}")
print(f" Current Price: {current_price}")
print(f" Product URL: {url}")
print("=" * 40 + "\n")
def monitor_price(url, threshold_price, history_file="ebay_prices.csv", proxy=None):
"""
Main price monitoring function
"""
print(f"[*] Checking product price... Target Threshold: {threshold_price}")
# 1. Retrieve page source
html = get_ebay_page(url, proxy=proxy)
if not html:
print("[Error] Unable to retrieve product page. Monitoring skipped.")
return
# 2. Parse product information
info = extract_product_info(html)
# 3. Save historical records
save_to_csv(info, product_url=url, filename=history_file)
# 4. Safely parse numeric price
try:
current_price = float(info["clean_price"])
except (ValueError, TypeError):
print(f"[Warning] Unable to parse price value. Original text: {info.get('raw_price')}")
return
print(
f" Successfully Retrieved -> "
f"[{info['title']}] Current Price: {current_price} "
f"(Original: {info['raw_price']})"
)
# 5. Threshold check
if current_price <= threshold_price:
send_alert(info["title"], current_price, url)
else:
print("[-] Target price not reached. Continue monitoring...")
# ----- Test Run -----
if __name__ == "__main__":
MY_PROXY = None # Example: "http://127.0.0.1:7890"
target_item_url = "https://www.ebay.com/itm/225643445557"
# Set target price to 500
monitor_price(
target_item_url,
threshold_price=500.0,
proxy=MY_PROXY
)
7. Automatically Send Price Alerts
Combined with the monitor_price function above, the following code enables email notifications through Gmail SMTP. You can also replace it with Slack Webhooks or enterprise messaging platforms:
import smtplib
from email.mime.text import MIMEText
from email.header import Header
def send_alert(
title,
price,
url,
sender="your@gmail.com",
receiver="your@gmail.com",
password="xxxx xxxx xxxx xxxx" # Google App Password
):
# 1. Build structured HTML email content
html_body = f"""
<html>
<body>
<h2 style="color: #e53238;">[eBay Price Alert Triggered]</h2>
<p><strong>Product:</strong> {title}</p>
<p><strong>Current Price:</strong>
<span style="color: #107a10; font-size: 18px; font-weight: bold;">
{price}
</span>
</p>
<p><strong>Product URL:</strong>
<a href="{url}" target="_blank">
View on eBay
</a>
</p>
<br>
<hr style="border:0; border-top:1px solid #eee;">
<p style="font-size: 12px; color: #999;">
This email was automatically generated by the eBay price monitoring script.
Please do not reply directly.
</p>
</body>
</html>
"""
# 2. HTML email format
msg = MIMEText(html_body, "html", "utf-8")
# 3. Proper UTF-8 encoding
msg["Subject"] = Header(
f"[eBay Price Alert] {title} dropped to {price}",
"utf-8"
)
msg["From"] = Header(
f"eBay Monitoring Bot <{sender}>",
"utf-8"
)
msg["To"] = Header(receiver, "utf-8")
try:
print("[*] Connecting to Gmail SMTP server...")
with smtplib.SMTP_SSL(
"smtp.gmail.com",
465,
timeout=15
) as server:
server.login(sender, password)
server.sendmail(
sender,
[receiver],
msg.as_string()
)
print("[Success] Price alert email sent successfully!")
return True
except smtplib.SMTPAuthenticationError:
print(
"[Error] Email sending failed: Authentication error. "
"Please verify your Google App Password."
)
except smtplib.SMTPConnectError:
print(
"[Error] Email sending failed: Unable to connect "
"to Gmail SMTP server."
)
except Exception as e:
print(f"[Error] Unknown email error: {e}")
return False
# ----- Test Run -----
if __name__ == "__main__":
test_title = "iPhone 15 Pro Max 256GB Unlocked"
test_price = "$899.00"
test_url = "https://www.ebay.com/itm/123456789"
# Replace sender, receiver and password before testing
# send_alert(test_title, test_price, test_url)
Note that Gmail requires Two-Step Verification to be enabled and an App Password to be generated before SMTP login can be used. Using your regular account password will result in authentication errors.
8. Schedule Automated Monitoring Tasks
Finally, use the schedule library to combine all components into an automated workflow that continuously runs in the background without manual intervention:
import os
import re
import csv
import time
import requests
from datetime import datetime
from bs4 import BeautifulSoup
import schedule
import smtplib
from email.mime.text import MIMEText
from email.header import Header
# ==================== 1. Global Configuration ====================
TARGET_URL = "https://www.ebay.com/itm/225643445557"
THRESHOLD_PRICE = 299.99
CHECK_INTERVAL_HOURS = 1
HISTORY_FILE = "ebay_prices.csv"
EMAIL_SENDER = "your@gmail.com"
EMAIL_RECEIVER = "your@gmail.com"
EMAIL_PASSWORD = "xxxx xxxx xxxx xxxx"
MY_PROXY = None
# Prevent duplicate alerts
HAS_ALERTED = False
# ==================== 2. Core Functions ====================
def get_ebay_page(url, proxy=None):
""" Safely retrieve page source """
session = requests.Session()
headers = {
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/124.0.0.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
}
session.headers.update(headers)
proxies = {"http": proxy, "https": proxy} if proxy else None
try:
response = session.get(
url,
proxies=proxies,
timeout=15
)
response.raise_for_status()
if "captcha" in response.url or "Robot Check" in response.text:
print(
f"[{datetime.now()}] "
f"[Warning] eBay bot verification triggered. Scraping failed."
)
return None
return response.text
except Exception as e:
print(
f"[{datetime.now()}] "
f"[Error] Network request exception: {e}"
)
return None
def extract_product_info(html):
""" Extract and clean product information """
soup = BeautifulSoup(html, "html.parser")
title_el = soup.find(
"h1",
class_="x-item-title__mainTitle"
)
title = (
title_el.get_text(strip=True)
if title_el else "Unknown Item"
)
price_el = soup.find(
"div",
class_="x-price-primary"
)
raw_price = (
price_el.get_text(strip=True)
if price_el else "N/A"
)
price_match = re.search(
r'\d+(?:\.\d+)?',
raw_price.replace(',', '')
)
clean_price = (
price_match.group(0)
if price_match else "N/A"
)
stock_el = soup.find(
"div",
class_="d-quantity__availability"
)
if not stock_el:
stock_el = soup.find(
class_=re.compile(r".*quantity__availability.*")
)
stock = (
stock_el.get_text(strip=True)
if stock_el else "Available"
)
return {
"title": title,
"raw_price": raw_price,
"clean_price": clean_price,
"stock": stock
}
def save_to_csv(data: dict, product_url, filename):
""" Append records to CSV """
file_exists = os.path.exists(filename)
try:
with open(
filename,
"a",
newline="",
encoding="utf-8-sig"
) as f:
writer = csv.DictWriter(
f,
fieldnames=[
"timestamp",
"title",
"price",
"stock",
"url"
]
)
if not file_exists:
writer.writeheader()
writer.writerow({
"timestamp":
datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
"title":
data["title"],
"price":
data["clean_price"],
"stock":
data["stock"],
"url":
product_url
})
except Exception as e:
print(f"[Error] Failed to write CSV: {e}")
def send_alert(title, price, url):
""" Send Gmail notification """
global HAS_ALERTED
if HAS_ALERTED:
print(
"[Info] Alert already sent during current low-price state. "
"Skipping duplicate notification."
)
return
subject = f"[eBay Price Alert] {title} dropped to {price}"
html_body = f"""
<html><body>
<h2 style="color:#e53238;">🚨 eBay Price Drop Alert</h2>
<p><strong>Product:</strong> {title}</p>
<p><strong>Current Price:</strong>
<span style="color:green;font-size:18px;font-weight:bold;">
{price}
</span></p>
<p><strong>Link:</strong>
<a href="{url}">View Product</a></p>
</body></html>
"""
msg = MIMEText(html_body, "html", "utf-8")
msg["Subject"] = Header(subject, "utf-8")
msg["From"] = Header(
f"eBay Monitoring Assistant <{EMAIL_SENDER}>",
"utf-8"
)
msg["To"] = Header(
EMAIL_RECEIVER,
"utf-8"
)
try:
with smtplib.SMTP_SSL(
"smtp.gmail.com",
465,
timeout=15
) as server:
server.login(
EMAIL_SENDER,
EMAIL_PASSWORD
)
server.sendmail(
EMAIL_SENDER,
[EMAIL_RECEIVER],
msg.as_string()
)
print("[Email] Price alert email sent successfully!")
HAS_ALERTED = True
except Exception as e:
print(f"[Email Error] Failed to send email: {e}")
# ==================== 3. Monitoring Task ====================
def monitor_price(url, threshold_price):
global HAS_ALERTED
current_time = datetime.now().strftime(
"%Y-%m-%d %H:%M:%S"
)
print(
f"\n[*] [{current_time}] Scheduled Check Triggered..."
)
try:
html = get_ebay_page(
url,
proxy=MY_PROXY
)
if not html:
return
info = extract_product_info(html)
save_to_csv(
info,
product_url=url,
filename=HISTORY_FILE
)
try:
current_price = float(
info["clean_price"]
)
except (ValueError, TypeError):
print(
f"[Warning] Unable to parse price: "
f"{info['raw_price']}"
)
return
print(
f" -> [{info['title']}] "
f"Current Price: {current_price} "
f"(Target: {threshold_price})"
)
if current_price <= threshold_price:
send_alert(
info["title"],
current_price,
url
)
else:
if HAS_ALERTED:
print(
"[Info] Price recovered above threshold. "
"Resetting alert status."
)
HAS_ALERTED = False
except Exception as top_e:
print(
f"[Crash Intercepted] Unknown error occurred: "
f"{top_e}. Waiting for next cycle..."
)
# ==================== 4. Scheduler ====================
if __name__ == "__main__":
print("=" * 40)
print(" Automated Price Monitoring System Started")
print(f" Monitoring Target: {TARGET_URL}")
print(f" Target Threshold: {THRESHOLD_PRICE}")
print(f" Check Frequency: Every {CHECK_INTERVAL_HOURS} hour(s)")
print("=" * 40)
# Run immediately at startup
monitor_price(
url=TARGET_URL,
threshold_price=THRESHOLD_PRICE
)
schedule.every(
CHECK_INTERVAL_HOURS
).hours.do(
monitor_price,
url=TARGET_URL,
threshold_price=THRESHOLD_PRICE
)
try:
while True:
schedule.run_pending()
time.sleep(1)
except KeyboardInterrupt:
print(
"\n[!] Stop signal received. "
"System exited safely."
)
Final Deployment Recommendations
● During testing, set THRESHOLD_PRICE higher than the current product price. For example, if the current price is $300, set the threshold to $400. This allows the alert system to trigger immediately so you can verify that email notifications are configured correctly.
● For long-term operation, ensure that your computer's automatic sleep or hibernation settings are disabled.
At this point, you have successfully built a complete eBay data scraping and price monitoring system.
III. FAQ
Q1: Why Do I Frequently Encounter 403 Errors or CAPTCHA Challenges?
eBay detects automated scraping through IP frequency analysis, User-Agent identification, and behavioral pattern analysis.
Possible solutions include:
● Using residential proxies such as IPFoxy Proxies to rotate IP addresses.
● Adding randomized request intervals between 1–5 seconds.
● Sending complete browser request headers.
● Using Selenium to simulate real browser behavior for more advanced scenarios.
Q2: What Should I Do If My Scraper Stops Working After eBay Updates Its Page Structure?
eBay periodically updates its HTML structure and CSS class names, which may cause existing selectors to fail.
Recommended practices include:
● Centralizing selector definitions for easier maintenance.
● Adding multiple fallback selectors for critical fields.
● Setting up error logging and alert mechanisms to detect scraping failures quickly.
Q3: How Can I Monitor Multiple Products Simultaneously?
Store product URLs and target price thresholds in a list or configuration file, then iterate through them within the scheduled monitoring task.
Best practices include:
● Adding random delays between requests.
● Rotating proxy IPs.
● Distributing requests across multiple sessions to reduce the likelihood of IP bans.
IV. Conclusion
For e-commerce sellers and operations teams, building a reliable eBay price monitoring system enables faster responses to market changes and supports more competitive pricing strategies. At the same time, using rotating residential proxies to reduce the risk of blocking is essential for maintaining long-term scraping stability.
If you want to further expand your system, consider integrating the eBay API for more reliable data access, incorporating machine learning models to predict price trends, or extending monitoring capabilities to other marketplaces such as Amazon and AliExpress to create a multi-platform pricing intelligence platform.
Top comments (0)