DEV Community: Noorsimar Singh

Proxy Routing for Web Scraping & Vibe Scraping in 2026: Best Proxies for Web Scraping (and Why Your Scraping Proxy Won’t Survive Prod)

Noorsimar Singh — Sun, 12 Jul 2026 06:31:45 +0000

If you have a web scraper that "works" locally and then collapses the moment you put it on a schedule, you are not alone. Most vibe scraping failures are not about selectors. They are about proxy choices, inconsistent web scraping proxies, and a proxy for web scraping setup that has zero opinion about risk, geo, sessions, or recovery. In other words: your IP strategy is the product, and your IP address is the first thing anti-bot systems score.

This guide is about proxy routing for web scraping. Not the generic "best proxies for web scraping" listicle stuff. Real routing: picking the right proxy provider, proxy type, proxy pool strategy, rotation strategy, and browser fingerprint profile per request.. then watching it like a dependency so you can fix it before your on-call gets spicy. If you came here looking for the best proxy or even the best proxy for web, the truth is you only get that answer by testing a real scraping proxy against your specific web pages.

Proxy for web scraping: the real problem is "proxy rotation" is not routing

Proxy rotation is a timer. Every N requests, swap IP address. It feels productive because you see new IPs in logs and you stop getting blocked for a few minutes.

Proxy routing is a policy. For each web request you decide: which proxy service, which geo, which ASN/ISP profile, whether you need sticky sessions for scraping, whether a browser is required, and what the fallback ladder looks like when you hit 403, 429, timeouts, or CAPTCHAs. Good routing also decides how to route scraping (HTTP vs browser vs API) and when to change proxy type, not just which IP you got.

Vibe web scraping breaks in production for a boring reason: it is optimized for "I got the data once," not SLOs, not cost, and definitely not incident recovery. Anti-bot systems learn patterns fast. If your proxy routing logic does not learn too, you just keep buying proxies and calling it scale.

Vibe scraping: what it looks like in the wild (and why it fails)

Symptoms you’ll recognize

403/429 rates spike right after a deploy, and nobody can reproduce it on their laptop.
CAPTCHAs show up after 5 to 10 minutes, not on the first request. That delay is a clue.
Sessions die mid-pagination, mid-checkout, or mid-login because your IP changes mid-flow (your IP address changed, but the cookies didn’t).
"Same code, different day" failures. You change nothing, but success rate tanks.
Proxy spend increases while throughput drops because retries silently double your request volume.

Root causes (anti-bot reality, not theory)

IP reputation is target-specific. The proxy network that works on one e-commerce site can be useless on another. One IP address can be "clean" for one target and burned for another.
ASN and ISP quality beat marketing labels. "Residential proxies" is a broad bucket. Some subnets are basically pre-burned. The same is true for datacenter proxies.
Session continuity matters. If the site ties cookies, TLS fingerprints, and behavior to an IP, naive proxy rotation breaks the chain.
Browser fingerprint and proxy alignment is real. A datacenter IP paired with a very human browser profile can look weirder than a bot.

If you want a quick gut-check from people who deal with this daily, the ethical residential proxy sourcing discussion is a useful thread. Not for vendor names. For the reality check on what "clean" IPs actually means (and why "best proxy" claims are often meaningless without context).

Using proxies for web scraping: the production definition of proxy routing

Routing decisions you should be making per request

Target domain policy: amazon.com is not walmart.com. Even different paths on the same host behave differently.
Geo requirement: country, region, sometimes city. Then validate it, because "US" is not always US.
Proxy type: datacenter proxies, ISP proxies, residential proxies, mobile proxies (different proxy types exist for a reason).
Session strategy: sticky sessions for scraping vs rotating proxies per request (and the right proxy rotation rules for each route).
Budget strategy: cheap-first for low-risk pages, success-first for high-value flows.
Fallback ladder: provider A to provider B to provider C, with explicit stop rules.
Browser mode: raw HTTP (via http proxies / https proxies), headless browser, or a rendering/scraping API.

The minimal routing metadata to log (non-negotiable)

target host and a path class (product page, search, category, login)
proxy provider and plan/config (residential and datacenter proxies, geo, session id)
egress IP and IP address (log what you actually used, not what you requested)
status code family plus block markers (captcha keywords, JS challenge hints)
latency percentiles per route (p50 and p95) and response size
retry count and retry reason
cost estimate per successful page (even if it’s rough)

Action step: add a request_id and propagate it through scraper logs, proxy logs, and your parsing output. If you cannot trace a single request across the pipeline, your debugging loop is going to be slow and expensive.. especially once you use proxies across multiple providers.

On teams that do this at scale, a monitoring layer helps. You can roll your own, but platforms like monitor your scrapers give you the "scraper health dashboard" view that many teams forget to build until after the third incident.

Proxy types for web scraping: a practical routing policy (start with a decision tree, then automate)

Decision tree (baseline)

Public, low-risk pages: start with datacenter proxies. Rotate per request. Keep concurrency high, but watch 429s.
Authenticated or session-heavy flows: use sticky sessions (ISP or residential). Limit concurrency per session.
Geo-sensitive content: enforce geo-targeted proxy rotation and track which regions actually work (verify the IP address periodically).
When blocks spike: escalate to residential proxies or mobile proxies, then add browser rendering if needed.

Example policy in pseudo-config (tool-agnostic)

Keep this as config, not code. That one decision alone saves you during incidents.

routes.yaml

example.com/search: primary datacenter_pool (rotate per request, 2 retries) then fallback residential_pool US (sticky 10m, 2 retries) then rendering_api / web scraping API (1 retry)
example.com/pdp: primary isp_pool US (sticky 30m, 3 retries) then fallback residential_pool (sticky 30m)

Where teams usually get it wrong

One global proxy pool for all targets. Everything shares the same failure domain and the same IP reputation bleed.
No separation between "cheap crawl" and "expensive extraction." Your costs balloon quietly.
No feedback loop. A route keeps failing, but the system keeps sending traffic into it.

If you want to centralize routes without binding yourself to one proxy vendor, using a proxy aggregator pattern is a clean approach. Some teams build it in-house. Others use something like the proxy aggregator so they can "combine all proxies into one api" and focus on policy, not plumbing. Either way, your proxy infrastructure should make provider swaps easy.

Scraping with proxies: dynamic proxy routing needs a feedback loop (not a static rule set)

Static rules are better than vibes. But dynamic proxy routing is where reliability shows up. You are basically doing traffic engineering, except the upstream is a hostile website and the downstream is your SLA. This is what proxies for scraping looks like when you’re scraping at scale.

Signals you should route on (with concrete thresholds)

403 rate over last 5 minutes > 3%: escalate proxy type or provider, and reduce concurrency.
429 spike: add jitter, slow down, and move to ISP/residential if the site rate-limits by IP address.
CAPTCHA detection > 1%: switch to a browser-based route or a rendering-capable scraping API (aka a web scraping api route).
p95 latency > 8s: quarantine that provider+geo combination temporarily.

Quarantine logic (simple, effective)

If provider+geo fails 5 times in a row, quarantine for 15 minutes.
Let it recover early if you see 2 consecutive successes during probe traffic.
Keep a known-good emergency route, even if it is expensive. Calm ops beats cheap ops.

In practice, provider failover is easier if your request code hits one endpoint and you change routing behind it. ScrapeOps offers both a Proxy API Aggregator and a residential proxy aggregator, and the "single integration, multiple providers" idea is the point. You can do the same with other stacks, but the pattern matters more than the vendor.

Proxy pool, rotating proxies, and geo: don’t just pick a country.. pick the right network shape

What "geo" really means operationally

Country is not region. Region is not city.
Some targets care about ASN or ISP more than country.
Localized SERPs and e-commerce pricing can vary by IP geolocation, timezone, language headers, currency, and login state.

Action steps: harden geo routing

Maintain a per-target geo matrix: allowed geos plus observed success rate.
Validate geo with an IP-check endpoint periodically. Do not trust "US" labels blindly; verify the egress IP and IP address match your expectation.
Keep separate proxy pool buckets: US-any, US-CA, EU-DE, APAC-SG, and so on. This becomes your web scraping proxy pool for different routes.

Linkable artifact: a geo matrix template

Copy this table into your runbook or wiki and actually fill it in. You will thank yourself later.

Columns: Target, Page Type, Required Geo, Preferred ASN/ISP, Proxy Type, Sticky?, Notes

If you are shopping for geo coverage, it helps to review how different providers describe their networks. For example, Bright Data publishes details about residential routing and coverage. Even if you do not buy, the documentation helps you ask better questions when choosing a web scraping proxy.

Proxy type deep dive: residential proxies vs datacenter proxies (the non-marketing version)

There is no universal best proxy. There is only "best proxy for web" for a specific target, page type, and traffic shape. That’s why proxy routing matters, and why different proxy types matter for different scraping tasks.

Datacenter proxies: when they’re the right tool

High-volume scraping where a small failure rate is acceptable.
Public pages with weak bot protection.
Cost-sensitive scraping projects where retries are controlled.

Residential proxies: when you actually need them

Targets with aggressive anti-bot scoring tied to IP reputation.
Price localization, inventory checks, or content that shifts by region.
Sites that block whole datacenter ranges fast.

ISP proxies: the boring but stable middle ground

Often more stable for sessions than rotating residential proxies.
Good for login flows, long pagination, and anything that breaks when IP changes.

Mobile proxies: expensive, situational

Use as a last resort or for mobile-only endpoints.
Only if your browser fingerprint and proxy alignment is tight.
Remember: mobile proxies route requests through carrier networks, so expect different latency and different block dynamics.

People love asking about cheap options. The problem is that cheap often means recycled IPs. If you want a reality check, the cheap residential proxies discussion is a good reminder that "cheap" and "reliable" rarely land in the same place for difficult targets. Be extra skeptical of free proxies for anything beyond trivial testing.

Proxy for web scraping sessions: sticky sessions (how to stop breaking workflows mid-flow)

What sticky sessions protect

Cart and session cookies tied to an IP address.
Anti-bot risk scoring that expects consistent network behavior.
Multi-step flows like pagination, filters, redirects, and login challenges.

Recommended session patterns (copy/paste)

Sticky by workflow: one session per job run. Great for long flows.
Sticky by entity: one session per account, category, or keyword bucket.
Sticky with TTL: 10 to 30 minutes, then rotate so you do not burn an IP into a known-bad state.

Failure mode checklist

If you see 302 loops, assume the session is flagged. Rotate session and clear cookies.
If CAPTCHAs increase with stickiness, your sessions are too "hot." Reduce TTL and concurrency per session.

Some proxy providers expose first-class sticky session controls, others make it awkward. If you already use a proxy management layer like Zyte's proxy manager, check how they handle session affinity and error transparency. Those details matter more than headline features from any proxy provider.

Web scraping proxies + fingerprinting: anti-bot evasion is incomplete without browser alignment

This is where teams lose time. They keep swapping proxy providers, but the actual problem is the browser layer. Or they fix the browser layer, but keep sending traffic through an IP range that the site already distrusts. Either way, you end up chasing ghosts and burning through proxy solutions that never had a chance.

Common mismatch patterns that trigger blocks

Datacenter IP with a "perfect" human Chrome profile and long-lived cookies. It looks curated.
Residential IP with headless signals, odd WebGL/canvas entropy, and inconsistent timezone.
Mobile user-agent with a desktop viewport running on a datacenter ASN.

Practical alignment rules

If you use Playwright, Puppeteer, or Selenium, align timezone, language, fonts, WebRTC behavior, and viewport.
If you route through mobile proxies, use realistic device metrics and network behavior.
Keep a fingerprint profile per route. Not per codebase. Your target cares about patterns, not your repo structure.

If you are generating scrapers quickly for new targets, an AI starter can save hours, but only if it respects production reality. Tools like the AI Scraper Builder can generate code across stacks and detect when JavaScript rendering is needed. Treat that output as a baseline, then wire it into your routing and monitoring so it survives real traffic and real web scraping projects.

Proxy server monitoring: proxy pool health (treat proxies like any other dependency)

A proxy server is not a magic tunnel. It is a dependency with failure modes. If you do not measure it, you will keep blaming your scraper. This is especially true once you’re running a proxy pool and switching between residential proxies, datacenter proxies, isp proxies, and mobile proxies.

The 4 metrics that matter

Success rate by provider, geo, and page type.
Block rate split by 403, 429, and CAPTCHA markers.
Latency percentiles (p50 and p95). Mean latency hides pain.
Cost per successful page including retries and rendering surcharges.

Alerts that reduce on-call pain (without noise)

Success rate drop above X% for 10 minutes on a critical target.
p95 latency breach plus backlog growth in your job queue.
Cost per success doubles week-over-week. That usually means routing drift or retries.

If you do not want to wire all of this into Grafana yourself, a managed view helps. ScrapeOps has a scraper health dashboard approach that pairs monitoring with scheduling, which is the combo most scraping teams end up needing anyway once jobs multiply.

Best proxies for web scraping: why the "best proxy" question is a trap (benchmark your target URL)

Generic proxy rankings are a comfort blanket. They get clicks, but they rarely predict your outcome. The "best proxies for web scraping" for a product page can be different from the best web scraping proxies for search pages on the same site. Different defenses. Different payload. Different risk scoring. That’s why the best web scraping is usually boring: define routes, measure, and iterate.

Why generic proxy rankings lie (even when well-intentioned)

Providers optimize for different verticals and traffic patterns.
One provider may crush PDP pages but fail on search and category pages.
Costs vary wildly once you add JS rendering, bandwidth, and retries.

A benchmark workflow you can run this week

Pick 3 representative URLs per target: search results, product/detail page, category/browse page.
Run 50 to 200 GET requests per provider configuration with consistent headers and pacing.
Record success rate, latency, CAPTCHA rate, and estimated cost per 1,000 successful pages.
Choose a primary and a fallback provider per page type, not per domain.

Shortcut: use a proxy benchmarking tool instead of building a harness

If you want a fast answer to "which provider works best for my target website", the proxy tester from ScrapeOps runs a live benchmark against your submitted URL and emails a report. It is not a static list. It is closer to how you actually buy proxies in the real world: test, compare, decide (and avoid buying proxies blind).

There is also a broader proxy comparison page that is useful when you are mapping features across proxy types for web scraping. Use it as a starting inventory, then trust benchmarks over opinions.

Proxy provider for web scraping: routing architecture patterns (from "script" to "scraping platform")

Pattern A: Single scraper to multi-provider routing gateway

Scraper sends requests to one internal endpoint.
Gateway decides provider, geo, session, retries, and backoff.
Pros: centralized control and consistent policies.
Cons: you must make it observable, or you just moved the problem.

Pattern B: Domain-based micro-policies (pragmatic, scalable)

Policy file per target domain and page type.
CI validates policy schema before deploy.
Rollback is a config revert, not a code redeploy.

Pattern C: Proxy aggregator plus scheduler plus health monitoring

Jobs grow. Targets grow. Ownership blurs. This is where systems get messy.
Either you build the platform yourself, or you adopt infrastructure that already exists.
ScrapeOps is one option here, especially if you want proxy management, scheduling, and monitoring in one place. Many teams stitch together Airflow or Temporal, vendor proxy APIs, and internal routing code.

If you like the aggregator approach but want to keep provider choice flexible, start with a single endpoint and grow from there. For example, ScrapeOps offers our proxy aggregator as a rotating proxy solution that can route across multiple proxy providers through one API. It is not the only way to do it, but it matches how production teams prefer to operate: fewer integration points, more control through policy.

Web scraper workflows: n8n-ready routing (scraping with proxies without building a control plane)

Lots of automation teams do not want to ship a whole new service just to manage scraping with proxies. Fair. If you are already living in n8n, you can still get 80% of the value by treating routing as a workflow step and by logging the egress IP and IP address for every critical web request.

A simple n8n flow for dynamic routing

Trigger: Cron or webhook.
Fetch URL node: call your routing endpoint or proxy API.
IF node: if 403/429/CAPTCHA markers show up, branch to an escalation route.
Parse node: extract and validate fields, then store the output.
Metrics: write route stats to Postgres, BigQuery, or ClickHouse.
Alert: Slack or email when thresholds breach.

What to persist per run (for debugging)

Chosen route and proxy provider metadata (including proxy type, geo, and session ID).
Retry chain history, not just the final outcome.
Output hash or record counts to catch silent partial failures.

If your pipeline includes structured extraction, consider separating "fetch" from "parse." For certain common targets, using a dedicated parser API can simplify downstream logic. ScrapeOps has a Parser API that turns raw HTML into structured JSON for supported sites, which can reduce the amount of brittle selector work in your automation layer.

Operational runbook: when the target changes their defenses overnight

Incident checklist (copy/paste)

Confirm it’s not your deploy. Roll back if needed.
Compare failures by page type. Search vs PDP vs category often diverge.
Check correlation: is it one provider, one geo, one proxy type, one IP range?
Switch to your fallback route and lower concurrency.
Add jitter and backoff. Stop hammering during a block event.
If you run a browser, rotate fingerprint profile and confirm alignment with network.
Post-incident: update the domain policy and add a regression test URL.

Regression tests that prevent repeat incidents

Daily canary per target: 10 to 20 requests, same URLs, same parse checks.
Alert on success rate drop and response-body anomalies, not just status codes.
Store a small HTML sample for diffing, but avoid sensitive data and credentials.

One quiet win here is having a fast way to generate a new baseline scraper when layouts change. If you want a starting point in Python or Node across frameworks, the ScrapeOps AI scraper generator can create editable code from a handful of sample URLs. It won’t replace engineering judgment, but it can shorten the "site changed, now what" loop for web scraping projects.

Cost control: proxy routing is mostly about retries you shouldn’t be doing

The math most teams ignore

If success rate drops from 95% to 80%, your retry volume does not just increase a bit. It can explode, depending on your retry policy. That means more proxy traffic, more bandwidth, higher latency, and a pipeline that misses deadlines.

Cost is not just "price per GB" or "price per request." It is cost per successful page, plus the engineering time spent babysitting a flaky route. The fastest way to reduce costs is usually to reduce wasteful web requests, not to chase a cheaper proxy plan.

Cost levers that don’t reduce reliability

Split crawl and extract tiers. Crawl can be cheap and fast. Extract can be slower, stickier, and safer.
Cache stable pages where possible (ETag, Last-Modified, or your own snapshot cache).
Stop retrying non-retryable failures. A consistent 404 is not a transient error.
Quarantine bad routes quickly. Do not pay to learn the same lesson 500 times.

If you are evaluating proxy providers mostly on price, take a step back and benchmark value. Some teams also like providers like Decodo for their residential proxies when they need coverage at a certain price point, but again, the right move is to test against your target, not your spreadsheet assumptions.

Scraping proxy providers and tools (non-exhaustive, pick what fits your ops model)

Proxy and scraping infrastructure options teams evaluate

Proxy networks and aggregators: Bright Data, Decodo, NodeMaven, ScrapeOps Proxy Aggregator.
Scraping APIs: Zyte API, ScraperAPI, ScrapingBee, Scrapfly (these often function as a scraping api + proxy network bundle).
Browser tooling: Playwright, Puppeteer, Selenium.
Extraction tooling: FireCrawl is popular for certain content extraction workflows, especially when you do not want to manage a full browser fleet.
AI-assisted scaffolding: tools that generate starter scrapers, like ScrapeOps AI Scraper Builder, can speed up new target onboarding.
Proxy protocols: choose the right mix of http proxies and https proxies depending on your targets and tooling.

Pick based on observability, failure handling, and integration effort. A flashy bypass feature is nice. A clear error taxonomy and predictable retries are nicer. For many scraping needs, proxies come in a few consistent forms (residential, datacenter, ISP, mobile), and proxies are best when you treat them as infrastructure.. not magic.

Copyable templates (designed to earn backlinks)

Template 1: Proxy routing policy skeleton

If you want one artifact to share internally, make it this. A small schema and strong defaults will push your team out of "random proxy rotation" and into actual proxy routing for web scraping.

match: host, page_type, and risk tier
primary route: provider, proxy type, geo, session mode
fallback routes: ordered ladder with stop conditions
thresholds: 403/429/CAPTCHA and latency triggers
quarantine: failure count, quarantine TTL, probe rules
fingerprint: profile id or settings pointer for browser routes

Template 2: Proxy provider evaluation checklist (choosing the right proxy)

Geo and ASN diversity: can you target specific regions and avoid poisoned subnets?
Block handling: what happens on 403/429? Do they expose real response bodies?
Session support: sticky session behavior, TTL, and whether it is consistent.
Observability: request IDs, per-request logs, error transparency.
Cost model clarity: rendering fees, bandwidth, retries, overage rules.
Dedicated proxies vs shared pools: when you need predictable session behavior, dedicated proxies or static residential proxies can be worth testing.
Residential proxy service sourcing: ask how the residential proxies use is obtained and what controls exist for abuse and churn.

Template 3: Benchmark spreadsheet columns

Provider and configuration
Target URL and page type
Success rate and CAPTCHA rate
Latency p50 and p95
Estimated cost per 1,000 successes
Notes: anomalies, ban patterns, failure signatures

If you want a structured way to compare providers without doing it manually, the ScrapeOps proxy comparison tool is handy as a catalog, and the proxy tester is the closer when you need results against a real target. When you’re comparing web scraping proxy providers, always evaluate a web scraping proxy type per page class, not just per domain.

FAQ

What’s the best proxy for web scraping in 2026?

The best proxy for web is the one that wins on your target URLs, with your traffic pattern, at your required geo.. and keeps that win rate stable over time. Static rankings age badly. Your target’s defenses change faster than blog posts do. That’s why "web scraping in 2026" is less about a single best proxy and more about routing, monitoring, and fast recovery.

Should I always use residential proxies?

No. Residential proxies are a tool, not a religion. Start with datacenter proxies when you can, then escalate by page type and risk. If your pipeline is well-routed, you use expensive routes only when they buy you success.

How many proxy providers should I use?

For most teams, two is enough for critical targets: a primary and a fallback. More providers can help, but only if you have routing and monitoring that can handle partial failures cleanly. This is where many scraping proxy providers look similar on paper, but differ in error transparency and stability.

How do I know if blocks are fingerprint vs IP?

If rotating proxies does not improve success rate, suspect fingerprint and behavior. If rotating fingerprints does not help, suspect IP reputation, ASN, or geo issues. Most real incidents are a mix, which is why proxy routing and browser fingerprint alignment belong in the same conversation.

Closing: if it can’t be observed and recovered, it’s not production-ready (web scraping in 2026)

Vibe scraping is fine for a prototype. Production web scraping is different. You need policies, dynamic routing, sticky sessions where they matter, geo that is validated, and proxy pool health monitoring that catches drift before your stakeholders do. Build a proxy network strategy that fits your scraping needs, not a "one proxy to rule them all" fantasy.

If you take one thing from this: stop asking "what are the best proxies for web scraping?" Start asking "what proxy routing for web scraping policy keeps success stable under partial failures, and what’s my rollback when the target changes?" That question leads to calm systems.. and it’s how you actually choose the best web scraping proxies for your own routes.

Building a Production-Ready Apple Store Reviews Scraper with Python & Scrapy

Noorsimar Singh — Thu, 17 Jul 2025 17:19:29 +0000

Extract app reviews from 15+ countries with robust error handling, anti-detection, and multi-format exports.

TL;DR

Built a scalable Apple Store reviews scraper using Python and Scrapy that automatically detects countries, handles rate limiting, and exports clean data. Features include proxy rotation, deduplication, and comprehensive error handling. Perfect for app market research and sentiment analysis.

The Problem That Started It All

Ever tried to analyze app reviews across different countries? I found myself manually copying data from Apple Store pages, which was... let's just say not my finest hour. The reviews were there, but getting them systematically was a nightmare.

Apple Store's dynamic content, rate limiting, and country-specific layouts made traditional scraping approaches fail spectacularly. I needed something that could handle the complexity while staying respectful of Apple's servers.

Why This Matters

App reviews are goldmines of user feedback, but they're scattered across different countries with varying languages and formats. For app developers, marketers, or researchers, having access to this data can mean the difference between a successful launch and a flop.

The challenge? Apple Store uses JavaScript-heavy pages, implements aggressive rate limiting, and serves different content based on your location. A simple requests + BeautifulSoup approach just won't cut it.

The Solution: A Robust Scrapy Spider

I built a production-ready scraper that handles all these challenges. Here's how it works:

1. Smart Country Detection

The scraper automatically detects the country from the Apple Store URL and configures the proxy accordingly:

def parse_app_url(self, url):
    """Parse Apple Store URL to extract country code and app ID"""
    try:
        parsed_url = urlparse(url)
        path_parts = parsed_url.path.split('/')

        # Extract country code (e.g., 'us', 'in', 'gb')
        if len(path_parts) >= 2:
            self.country_code = path_parts[1].lower()

        # Extract app ID from the path
        for part in path_parts:
            if part.startswith('id') and part[2:].isdigit():
                self.app_id = part[2:]  # Remove 'id' prefix
                break

This means you can scrape the same app from different countries without changing any code. The scraper supports 15+ countries including US, India, UK, Canada, Australia, and more.

2. Robust Data Extraction with Fallbacks

Apple Store's HTML structure can be... unpredictable. I implemented multiple fallback strategies:

def extract_app_info(self, response):
    """Extract app information with robust fallbacks"""
    # 1. Try meta og:title
    app_name = response.css('meta[property="og:title"]::attr(content)').get()

    # 2. Try JSON-LD block
    json_ld = response.css('script[type="application/ld+json"]::text').get()
    if not app_name and json_ld:
        try:
            data = json.loads(json_ld)
            app_name = data.get('name')
        except Exception:
            pass

    # 3. Try <title> tag
    if not app_name:
        app_name = response.css('title::text').get()
        if app_name:
            app_name = app_name.replace(' - Ratings and Reviews', '').strip()

This approach ensures we get the app name even if Apple changes their HTML structure.

3. Anti-Detection Features

The scraper includes several features to avoid being blocked:

User-agent rotation: Different browsers and devices
Random delays: 2-3 seconds between requests
Proxy rotation: Using ScrapeOps for reliable IP rotation
JavaScript rendering: Handles dynamic content loading

4. Production-Ready Data Pipeline

I implemented a comprehensive pipeline system:

ITEM_PIPELINES = {
    'appstore_reviews_scraper.pipelines.AppStoreReviewsDeduplicationPipeline': 400,
    'appstore_reviews_scraper.pipelines.AppStoreReviewsLoggingPipeline': 500,
    'appstore_reviews_scraper.pipelines.AppStoreReviewsJsonPipeline': 600,
    'appstore_reviews_scraper.pipelines.AppStoreReviewsCsvPipeline': 700,
}

This handles deduplication, logging, and exports to both JSON and CSV formats.

Getting Started

The setup is straightforward:

Clone the repository:

   git clone https://github.com/Simple-Python-Scrapy-Scrapers/appstore-reviews-scrapy-scraper.git
   cd appstore-reviews-scrapy-scraper

Create a virtual environment:

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

Install dependencies:

   pip install -r requirements.txt

Grab a free ScrapeOps API key (this saved me hours of proxy setup)
Configure your API key in appstore_reviews_scraper/settings.py:

   SCRAPEOPS_API_KEY = 'YOUR_API_KEY'

Running the Scraper

Now for the fun part - actually running the scraper. I use Scrapy commands directly for better control and debugging:

Basic Usage

# Scrape 50 reviews from Instagram US
scrapy crawl appstore_reviews -a app_url="https://apps.apple.com/us/app/instagram/id389801252" -a max_reviews=50

# Scrape 100 reviews from TikTok India
scrapy crawl appstore_reviews -a app_url="https://apps.apple.com/in/app/tiktok/id835599320" -a max_reviews=100

# Scrape 200 reviews from WhatsApp UK
scrapy crawl appstore_reviews -a app_url="https://apps.apple.com/gb/app/whatsapp-messenger/id310633997" -a max_reviews=200

Advanced Usage

# Scrape with custom settings
scrapy crawl appstore_reviews -a app_url="https://apps.apple.com/us/app/spotify-music/id324684580" -a max_reviews=75 -s DOWNLOAD_DELAY=3

# Enable debug logging
scrapy crawl appstore_reviews -a app_url="https://apps.apple.com/ca/app/netflix/id363590051" -a max_reviews=150 -s LOG_LEVEL=DEBUG

# Run with specific output format
scrapy crawl appstore_reviews -a app_url="https://apps.apple.com/us/app/instagram/id389801252" -a max_reviews=25 -s FEEDS='{"data/reviews.json": {"format": "json"}}'

What Happens When You Run It

Country Detection: The scraper automatically detects the country from the URL
Proxy Configuration: Sets up ScrapeOps proxy for that specific country
Data Extraction: Extracts app info and reviews with fallback strategies
Pipeline Processing: Deduplicates, validates, and exports data
File Generation: Creates timestamped JSON and CSV files in the data/ directory

What You Get

The scraper extracts comprehensive data:

App Information:

App ID, name, developer
Category, pricing, overall rating
Size, version, compatibility
Age rating, languages supported

Review Data:

Reviewer name and rating (1-5 stars)
Review title and full text
Review date and helpful votes
Country-specific metadata

Sample Output

The scraper generates files like:

apps_20250714_183332.json - App information
reviews_20250714_183332.csv - Review data in CSV format
reviews_20250714_183332.json - Review data in JSON format

Advanced Features

Multi-Country Support

The scraper automatically detects and handles different Apple Store regions:

# US Instagram reviews
scrapy crawl appstore_reviews -a app_url="https://apps.apple.com/us/app/instagram/id389801252" -a max_reviews=100

# Indian TikTok reviews  
scrapy crawl appstore_reviews -a app_url="https://apps.apple.com/in/app/tiktok/id835599320" -a max_reviews=100

# UK WhatsApp reviews
scrapy crawl appstore_reviews -a app_url="https://apps.apple.com/gb/app/whatsapp-messenger/id310633997" -a max_reviews=100

Custom Settings Override

You can override any Scrapy setting on the command line:

# Faster scraping (use with caution)
scrapy crawl appstore_reviews -a app_url="..." -a max_reviews=50 -s DOWNLOAD_DELAY=1 -s CONCURRENT_REQUESTS=3

# More conservative scraping
scrapy crawl appstore_reviews -a app_url="..." -a max_reviews=50 -s DOWNLOAD_DELAY=5 -s CONCURRENT_REQUESTS=1

Debug Mode

When things go wrong (and they will), enable debug mode:

scrapy crawl appstore_reviews -a app_url="..." -a max_reviews=5 -s LOG_LEVEL=DEBUG

This will show you exactly what's happening, including the HTML responses saved to the data/ directory for analysis.

The Technical Deep Dive

How the Spider Works

The spider follows this flow:

Start Request: Parses the app URL to extract country and app ID
Proxy Setup: Configures ScrapeOps proxy for the detected country
Page Fetching: Downloads the reviews page with JavaScript rendering
Data Extraction: Uses robust selectors with multiple fallbacks
Pipeline Processing: Deduplicates and exports data
Pagination: Follows next page links if more reviews are needed

Key Technical Decisions

Why Scrapy over requests?

Built-in retry mechanisms
Automatic request queuing
Pipeline system for data processing
Better handling of concurrent requests

Why multiple fallback strategies?

Apple Store's HTML structure changes frequently
Different countries may have slightly different layouts
Ensures robustness against future changes

Why ScrapeOps proxy?

Handles JavaScript rendering automatically
Country-specific proxy rotation
Built-in rate limiting and retry logic
Reliable and scalable

Learning Resources

While building this scraper, I found these resources incredibly helpful:

Apple Website Analyzer: Comprehensive analysis of Apple Store's scraping challenges, legal considerations, and technical details
How to Scrape Apple Store Guide: Step-by-step tutorials and best practices

These helped me understand the nuances of Apple Store scraping and implement the right strategies.

Why ScrapeOps Made the Difference

I initially tried building my own proxy rotation system. It was... not fun. After grabbing a free ScrapeOps API key, the whole process became much smoother. Their proxy service handles:

Country-specific proxies: Automatically matches the target country
JavaScript rendering: Handles dynamic content without additional setup
Rate limiting: Built-in delays and retry mechanisms
Analytics: Monitor your scraping performance

Troubleshooting Common Issues

"No review containers found"

This usually means Apple changed their HTML structure or the proxy didn't return the expected content.

Solution: Check the saved HTML files in the data/ directory and update selectors if needed.

"Proxy connection failed"

Your ScrapeOps API key might be invalid or you've exceeded your quota.

Solution: Verify your API key and check your ScrapeOps dashboard for usage.

"Rate limited"

You're making requests too quickly.

Solution: Increase DOWNLOAD_DELAY in your command:

scrapy crawl appstore_reviews -a app_url="..." -a max_reviews=50 -s DOWNLOAD_DELAY=5

Next Steps

The scraper is production-ready, but here are some enhancements you could add:

Sentiment analysis: Process review text for sentiment scores
Database integration: Store data in PostgreSQL or MongoDB
Scheduling: Set up automated scraping with cron jobs
API endpoint: Create a REST API for the scraper
Dashboard: Build a web interface for monitoring and analysis

Wrapping Up

Building this scraper taught me that modern web scraping is less about brute force and more about understanding the target website's architecture. Apple Store's dynamic content and anti-bot measures require a thoughtful approach.

The key was combining robust error handling with intelligent fallbacks, while staying respectful of Apple's servers. The result is a scraper that's both reliable and maintainable.

Check out the full code on GitHub and let me know if you build something interesting with it. If you found this helpful, consider starring the repository - it helps other developers discover useful tools like this.

Title Variations

Variation 1: Problem-Solution Focus

"How I Built a Scalable Apple Store Reviews Scraper That Actually Works"

Variation 2: Technical Deep-Dive

"Building a Production-Ready Web Scraper: Lessons from Apple Store"

Variation 3: Educational Approach

"From Manual Copy-Paste to Automated Scraping: A Python Journey"

Variation 4: Results-Oriented

"Extract Apple Store Reviews from 15+ Countries with This Python Scraper"

G2.com Scraping: Handling Anti-Bot Measures with Scrapy

Noorsimar Singh — Mon, 14 Jul 2025 17:00:07 +0000

Building a Production-Ready G2.com Scraper with Python and Scrapy

Learn how to build a robust web scraper for G2.com that handles anti-bot measures, exports clean data, and scales from development to production.

TL;DR

Built a G2.com scraper using Scrapy that extracts category listings and product reviews. Features include anti-bot detection, proxy rotation via ScrapeOps, duplicate handling, and clean CSV/JSON exports. Perfect for market research and competitive analysis.

I recently needed to gather competitive intelligence from G2.com for my project. What started as a simple script quickly evolved into a production-ready scraper that handles G2's anti-bot measures, exports clean data, and scales from development to production. Here's how I built it and what I learned along the way.

Why This Matters

G2.com is a goldmine for B2B market research, but it's also protected by sophisticated anti-bot measures. Most scraping attempts fail due to rate limiting, IP blocking, or JavaScript challenges. A robust scraper needs to handle these obstacles while maintaining data quality and respecting the site's policies.

The Architecture

The scraper uses two main spiders:

Category Spider: Extracts product listings from category pages
Product Reviews Spider: Collects detailed reviews from individual product pages

Both spiders share a common pipeline for data validation, duplicate removal, and export formatting.

Core Implementation

1. Universal Selector Discovery

G2's layout varies across pages, so I implemented a fallback system that tries multiple selectors:

def discover_review_containers(self, response):
    container_selectors = [
        'article.elv-bg-neutral-0',
        'article[data-testid*="review"]',
        'div[itemprop="review"]',
        # ... more fallbacks
    ]

    for selector in container_selectors:
        containers = response.css(selector)
        if containers and self._verify_review_content(containers):
            return containers
    return []

This approach ensures the scraper adapts to different page layouts without manual intervention.

2. Anti-Bot Middleware Stack

The middleware pipeline handles various anti-bot measures:

DOWNLOADER_MIDDLEWARES = {
    'g2_scraper.middlewares.RandomUserAgentMiddleware': 400,
    'g2_scraper.middlewares.RandomDelayMiddleware': 401,
    'g2_scraper.middlewares.AntiBotDetectionMiddleware': 402,
    'scrapeops_scrapy_proxy_sdk.ScrapeOpsScrapyProxySdk': 725,
}

The RandomUserAgentMiddleware rotates through realistic browser signatures, while the RandomDelayMiddleware adds natural timing variations.

3. Smart Duplicate Detection

Instead of simple field matching, the duplicate pipeline creates unique identifiers:

def process_item(self, item, spider):
    adapter = ItemAdapter(item)

    if 'reviewer_name' in adapter and 'review_date' in adapter:
        # Reviews: reviewer + date + review snippet
        review_text = adapter.get('review_text', '')[:50]
        item_id = f"review_{adapter.get('reviewer_name')}_{adapter.get('review_date')}_{review_text}"
    elif 'product_name' in adapter and 'product_url' in adapter:
        # Products: name + URL
        item_id = f"product_{adapter.get('product_name')}_{adapter.get('product_url')}"

    if item_id in self.ids_seen:
        raise DropItem(f"Duplicate item found: {item_id}")
    else:
        self.ids_seen.add(item_id)
        return item

This prevents duplicate reviews while allowing multiple products with the same name.

4. Dynamic Export Pipeline

The export pipeline creates files only for item types that actually have data:

def process_item(self, item, spider):
    # Determine item type
    if 'reviewer_name' in adapter:
        item_type = 'review'
    elif 'product_name' in adapter:
        item_type = 'product'
    elif 'category_name' in adapter:
        item_type = 'category'

    # Create file only when first item of this type is encountered
    if item_type not in self.active_types:
        filename = f'data/g2_{item_type}s_{timestamp}.csv'
        # ... create file and write header
        self.active_types.add(item_type)

    # Write data row
    csv_writer.writerow(clean_item_obj.values())

This prevents empty files and keeps the output directory clean.

Usage Examples

Category Scraping

scrapy crawl g2_category -a category=system-security -a limit=5

This extracts the top 5 products from the system-security category, including ratings, review counts, and vendor information.

Product Reviews Scraping

scrapy crawl g2_product_reviews -a product_url="https://www.g2.com/products/rollworks-account-based-platform/reviews"

This collects detailed reviews with pros/cons, reviewer information, and ratings.

Key Features That Made This Production-Ready

JavaScript Rendering: Uses render_js=true parameter for fully rendered pages
Proxy Rotation: Integrated ScrapeOps proxy for IP rotation and geolocation
Data Validation: Automatic cleaning and validation of all scraped fields
Error Recovery: Exponential backoff and retry mechanisms
Rate Limiting: Respectful delays and auto-throttling

What I Learned

The biggest challenge was handling G2's dynamic content loading. Initially, I tried static selectors, but they failed when the site updated its layout. The universal selector discovery system solved this by trying multiple patterns and verifying content before proceeding.

Another key insight was the importance of proper duplicate detection. Simple field matching caused issues when the same reviewer posted multiple reviews for the same product. The current approach uses a combination of reviewer name, date, and review snippet to create truly unique identifiers.

Getting Started

Clone the repository: g2-scrapy-scraper
Install dependencies: pip install -r requirements.txt
Grab a free ScrapeOps API key for proxy rotation
Run the spiders: Use the commands above

Next Steps and Resources

For deeper insights into G2's scraping challenges, check out the G2 Website Analyzer which covers anti-bot measures, legal considerations, and technical challenges.

If you need step-by-step guidance, the How-to Scrape G2 Guide provides detailed walkthroughs for various scraping scenarios.

Why ScrapeOps Made a Difference

I initially tried building this with free proxies, but the success rate was abysmal. After grabbing a free ScrapeOps API key, the success rate jumped to 95%+. The proxy rotation and geolocation features eliminated most blocking issues, while the monitoring dashboard helped me optimize the scraping strategy.

Conclusion

This scraper demonstrates how to build production-ready web scraping solutions that handle real-world challenges. The modular architecture makes it easy to extend for other sites, while the robust error handling ensures reliable operation.

The complete code is available on GitHub - feel free to star it if you find it useful, and let me know if you have questions or suggestions for improvements.

Scraping eBay's 1.2 Billion Listings: A Scrapy Tale

Noorsimar Singh — Fri, 11 Jul 2025 17:21:45 +0000

TL;DR: Built a production-ready eBay scraper using Scrapy framework with ScrapeOps proxy integration. Handles search results, product details, price extraction, and exports clean CSV/JSON data while bypassing anti-bot measures.

Last quarter, I needed to track laptop prices across eBay for a friend research project. What started as a "quick scraping script" turned into a full-featured eBay scraper that I'm now sharing as an open-source project.

The challenge? eBay has sophisticated anti-bot protection, dynamic pricing, and multiple page layouts. The solution? A robust Scrapy spider with proper proxy rotation and intelligent data extraction.

Why eBay Scraping Matters for Developers

E-commerce data extraction is everywhere in modern development. Whether you're building price comparison tools, market research platforms, or inventory management systems, you'll eventually need to scrape product data.

eBay presents unique challenges:

Multiple anti-detection systems
Complex auction vs. fixed-price listings
Global marketplace variations
Real-time price changes

Getting this right teaches you patterns that apply to almost any web scraping project.

The Architecture: Two Spiders, Clean Data

I built two specialized spiders instead of one monolithic scraper:

1. Search Spider (`ebay_search.py`)

Extracts product listings from search results pages:

def extract_search_item_data(self, container, meta):
    """Extract data from a single search result item"""
    item = EbaySearchItem()

    # Skip placeholder items (common eBay issue)
    if not item['product_id'] or not item['product_id'].isdigit() or len(item['product_id']) < 9:
        return None

    # Price extraction with fallbacks
    price_element = response.css('.s-item__price .notranslate::text').get()
    if not price_element:
        price_element = response.css('.s-item__price::text').get()
    item['current_price'] = price_element.strip() if price_element else ''

    return item

2. Product Spider (`ebay_product.py`)

Extracts detailed information from individual product pages:

def extract_product_data(self, response):
    """Extract detailed product data from eBay product page"""
    # Updated selectors for new eBay layout
    price_element = response.css('.x-price-primary .ux-textspans::text').get()
    if not price_element:
        price_element = response.css('.x-price-approx__price .ux-textspans::text').get()

    # Handle multiple currencies (AU $575.00, $199.99, etc.)
    item['current_price'] = price_element.strip() if price_element else ''

The key insight? eBay frequently updates their CSS selectors, so I implemented cascading fallbacks that check multiple possible selectors.

Handling Anti-Bot Protection with ScrapeOps

Here's where most DIY scrapers fail. eBay's security is no joke. I integrated ScrapeOps proxy rotation early in the project, and honestly, it saved me weeks of debugging.

# settings.py
SCRAPEOPS_API_KEY = 'YOUR_API_KEY'
SCRAPEOPS_PROXY_ENABLED = True
SCRAPEOPS_RENDER_JS = True
SCRAPEOPS_PROXY_SETTINGS = {'bypass': 'generic_level_2'}

DOWNLOADER_MIDDLEWARES = {
    'scrapeops_scrapy_proxy_sdk.scrapeops_scrapy_proxy_sdk.ScrapeOpsScrapyProxySdk': 725,
}

The bypass: generic_level_2 parameter was crucial for handling eBay's JavaScript challenges. Without it, I was getting blocked after just a few requests.

Clean Data Export Pipeline

Raw scraped data is messy. I built validation and cleaning pipelines that output production-ready CSV and JSON:

# pipelines.py
class EbayDataCleaningPipeline:
    def process_item(self, item, spider):
        # Normalize prices across different currencies
        if item.get('current_price'):
            item['current_price'] = self.clean_price(item['current_price'])

        # Standardize conditions
        if item.get('condition'):
            item['condition'] = self.normalize_condition(item['condition'])

        return item

The result? Clean CSV files with consistent formatting, ready for analysis or database import.

Running the Scraper

Basic usage is straightforward:

# Search for products
scrapy crawl ebay_search -a query="laptop" -a max_results=50

# Get detailed product info  
scrapy crawl ebay_product -a item_ids="123456789,987654321"

# Export to specific format
scrapy crawl ebay_search -o results.json -a query="vintage camera"

The scraper automatically handles:

Rate limiting and delays
User agent rotation
Failed request retries
Data validation and cleaning

Next Steps and Enhancements

Want to take this further? The complete GitHub repository includes advanced features like:

Multi-currency price normalization
Seller reputation analysis
Historical price tracking setup
Global eBay site support (UK, DE, CA, AU)

For deeper insights into eBay's anti-bot measures and scraping strategies, I recommend checking out the eBay Website Analyzer and their comprehensive scraping guide.

The ScrapeOps Integration Experience

Full transparency: I grabbed a free ScrapeOps API key early in this project, and it eliminated the proxy headaches entirely. Their free tier covers 1,000 requests monthly, which is perfect for development and testing.

The JavaScript rendering feature was particularly valuable for eBay's dynamic content. Instead of wrestling with Selenium or Playwright, I got reliable results with a simple configuration change.

Wrapping Up

Building a production-ready web scraper taught me that the hard part isn't extracting data—it's handling all the edge cases, anti-bot measures, and data quality issues that come up in real-world usage.

This eBay scraper handles those challenges with proven patterns you can apply to other e-commerce sites. The code is open source, well-documented, and battle-tested.

Found this useful? Star the GitHub repo and let me know what you build with it. I'm always curious to see how other developers adapt these patterns for their own projects.

What's your biggest web scraping challenge? Drop a comment below—I might cover it in a future post.

Building a YouTube Scraper with Scrapy: A Developer's Journey

Noorsimar Singh — Thu, 10 Jul 2025 12:03:22 +0000

Extract video data, channel analytics, and search results from YouTube using Python and Scrapy with proxy rotation and anti-bot protection.

TL;DR

Built a production-ready YouTube scraper that extracts video search results and channel analytics using Scrapy. Features include proxy rotation via ScrapeOps, data validation pipelines, and CSV/JSON export. Perfect for researchers, marketers, and developers who need reliable YouTube data extraction without hitting API limits.

The Problem That Started It All

Last year, I was working on a content analysis project and needed to gather YouTube data at scale. The official YouTube Data API has strict quotas, and manual data collection was taking forever. So I decided to build something better—a robust scraper that could handle YouTube's anti-bot measures while extracting clean, structured data.

What I ended up with was a comprehensive YouTube scraper that not only works reliably but also teaches some valuable lessons about modern web scraping workflows.

Why YouTube Scraping Matters

YouTube contains a goldmine of data for content creators, marketers, and researchers. But accessing this data programmatically comes with challenges:

API Limitations: YouTube Data API has strict quotas and rate limits
Anti-Bot Measures: YouTube actively detects and blocks automated requests
Data Quality: Raw HTML extraction often yields inconsistent results
Scale Issues: Manual data collection doesn't scale for research projects

The solution? A well-architected Scrapy-based scraper with proper proxy rotation and data validation.

The Architecture

The scraper follows a clean, modular design with separate spiders for different use cases:

# Core spiders
youtube_search.py      # Extract video search results
youtube_channel.py     # Gather channel analytics

Each spider yields structured data through Scrapy's item pipeline system, ensuring clean, validated output.

Key Features That Make It Work

1. Smart Data Extraction

The scraper uses multiple extraction strategies to handle YouTube's dynamic content:

# Primary: Extract from ytInitialData JavaScript objects
def parse_search_results(self, response):
    # Parse YouTube's internal data structure
    yt_data = self.extract_yt_initial_data(response)

    # Fallback: HTML parsing when JS extraction fails
    if not yt_data:
        return self.parse_html_fallback(response)

2. Proxy Rotation with ScrapeOps

To avoid IP blocking, the scraper integrates with ScrapeOps for proxy rotation:

# settings.py
SCRAPEOPS_API_KEY = 'your_api_key_here'
SCRAPEOPS_PROXY_ENABLED = True

DOWNLOADER_MIDDLEWARES = {
    'scrapeops_scrapy_proxy_sdk.scrapeops_scrapy_proxy_sdk.ScrapeOpsScrapyProxySdk': 725,
}

I grabbed a free ScrapeOps API key during development, and it made a huge difference in reliability. The free tier gives you 1,000 requests —perfect for testing and small projects.

3. Data Validation Pipeline

Every scraped item goes through a validation pipeline:

class YoutubeDataValidationPipeline:
    def process_item(self, item, spider):
        # Validate required fields
        required_fields = {
            'YoutubeSearchItem': ['video_url', 'title', 'channel_name'],
            'YoutubeChannelItem': ['channel_name', 'channel_url']
        }

        # URL validation and data cleaning
        if not self._is_valid_youtube_url(adapter.get('video_url')):
            raise DropItem(f"Invalid YouTube URL")

4. Flexible Export System

The scraper exports data in multiple formats with dynamic field handling:

# CSV export with automatic field detection
FEEDS = {
    'data/youtube_search_results.csv': {'format': 'csv'},
    'data/youtube_channels.csv': {'format': 'csv'},
    'data/youtube_videos.csv': {'format': 'csv'}
}

Usage Examples

Search for Videos

# Basic search
scrapy crawl youtube_search -a query="python programming" -a max_results=50

# Advanced search with filters
scrapy crawl youtube_search -a query="machine learning" -a max_results=100

Extract Channel Data

# Scrape by channel handles
scrapy crawl youtube_channel -a channel_handles="@freecodecamp,@programmingwithmosh"

# Scrape by URLs
scrapy crawl youtube_channel -a channel_urls="https://www.youtube.com/@scrapeops"

What You Get

The scraper extracts comprehensive data:

Search Results:

Video URLs, titles, descriptions, thumbnails
Channel information and verification status
View counts (normalized to integers)
Upload dates and duration
Content type detection

Channel Analytics:

Profile pictures and banner images
Subscriber counts and video counts
Channel descriptions and social links
Performance metrics and engagement rates

Data Analysis Ready

The exported CSV files are ready for analysis:

import pandas as pd

# Load scraped data
search_data = pd.read_csv('data/youtube_search_results.csv')
channel_data = pd.read_csv('data/youtube_channels.csv')

# Analyze view distribution
search_data['views_normalized'].hist(bins=50)
plt.title('Distribution of Video Views')
plt.show()

Lessons Learned

Building this scraper taught me several important lessons about modern web scraping:

Proxy rotation is essential - Without it, you'll get blocked quickly
Data validation matters - Clean data saves hours of post-processing
Modular design pays off - Separate spiders for different use cases
Error handling is crucial - YouTube's structure changes frequently

Next Steps and Resources

If you're interested in diving deeper into YouTube scraping, I recommend checking out the YouTube Website Analyzer for insights into scraping difficulty and legal considerations. There's also a comprehensive step-by-step guide that covers advanced techniques.

Get Started

The complete code is available on GitHub: youtube-scrapy-scraper

To get started:

Clone the repository
Install dependencies: pip install -r requirements.txt
Grab a free ScrapeOps API key
Update your API key in settings.py
Run your first scrape

Wrapping Up

This YouTube scraper represents what modern web scraping should be: reliable, scalable, and maintainable. It's not just about extracting data—it's about building robust systems that can handle real-world challenges.

The project is open-source and actively maintained. If you find it useful, consider starring the repository or contributing to make it even better. And if you run into any issues, the community is always there to help.

Happy scraping!

Pinterest Scraping in 2025: Why I Built a Production-Ready Scrapy Spider (And You Should Too)

Noorsimar Singh — Wed, 09 Jul 2025 16:51:12 +0000

The Problem That Changed My Perspective

Last month, I was helping a client analyze visual content trends for their e-commerce brand. They needed to understand what home decor pins were gaining traction, which boards were most influential, and how their competitors were performing on Pinterest.

Traditional social media analytics tools? They barely scratched Pinterest's surface. Manual research? Absolutely not scalable.

That's when I realized: Pinterest's visual data goldmine is largely untapped by developers.

Why Pinterest Scraping is Different in 2025

Pinterest isn't just another social platform—it's a visual search engine with over 450 million monthly active users. But here's the catch: scraping Pinterest effectively requires understanding its unique challenges:

The Technical Reality Check

Heavy JavaScript rendering (goodbye, simple HTTP requests)
Sophisticated anti-bot detection (IP blocking is real)
Dynamic content loading (infinite scroll complexity)
Rate limiting that actually works (they mean business)

I learned this the hard way when my first attempts resulted in blocked IPs and empty responses. The Pinterest Website Analyzer confirmed what I suspected: this platform has some serious scraping challenges.

Building a Pinterest Scraper That Actually Works

After weeks of trial and error, I built a production-ready Pinterest scraper using Scrapy. Here's what I discovered works:

The Three-Spider Architecture

Instead of one monolithic scraper, I created three specialized spiders:

Pinterest Pins Spider - Individual pin data with engagement metrics
Pinterest Boards Spider - Board analytics and user insights
Pinterest Search Spider - Search results and trending content

Each spider handles different data structures and extraction patterns, making the entire system more robust.

JavaScript Rendering (The Game Changer)

Pinterest's content is heavily JavaScript-dependent. Using ScrapeOps proxy with JavaScript rendering was crucial:

# This approach actually works
proxy_url = f"https://proxy.scrapeops.io/v1/?api_key={api_key}&url={search_url}&render_js=true&wait=3000"

The 3-second wait time ensures all dynamic content loads properly.

Real Success Metrics

After optimization, my scraper achieves:

95%+ request success rate
25-50 pins per search query
15-20 boards per category
Clean CSV exports with timestamped data

The Data Goldmine You're Missing

Here's what you can extract with a properly built Pinterest scraper:

Pin-Level Intelligence

Engagement metrics (likes, comments, repins)
Source attribution and link analysis
Visual content metadata
Shopping data and pricing trends

Board Analytics

Follower counts and growth patterns
Collaboration networks
Content category analysis
Seasonal trend identification

Search Insights

Trending topics and hashtags
Content discovery patterns
Competitive intelligence
Market sentiment analysis

Technical Implementation Insights

Proxy Rotation is Non-Negotiable

Pinterest's anti-bot measures are sophisticated. I found that using ScrapeOps proxy rotation was essential for maintaining consistent access. Their free tier (1,000 requests) is perfect for development and testing.

The CSV Pipeline Strategy

Rather than complex databases, I optimized for CSV output with smart filtering:

# Only create files for relevant data types
non_empty_item = {k: v for k, v in cleaned_item.items() 
                  if v and v != '' and v != '0' and v != 'false'}

This approach keeps data clean and immediately usable for analysis.

Rate Limiting That Respects the Platform

My configuration respects Pinterest's resources:

CONCURRENT_REQUESTS = 1
DOWNLOAD_DELAY = 2
RANDOMIZE_DOWNLOAD_DELAY = 0.5

Ethical scraping isn't just good practice—it's sustainable scraping.

Real-World Applications I've Seen

E-commerce Trend Analysis

A fashion brand used the scraper to identify emerging style trends 3 months before they peaked. They adjusted their inventory accordingly and saw 23% higher sales on trending items.

Content Marketing Intelligence

A home decor blog analyzed competitor pin performance to optimize their content strategy. They identified underserved niches and increased their Pinterest traffic by 180%.

Market Research at Scale

A startup used board analytics to understand consumer preferences in their target demographic, informing product development decisions.

The Resources That Made the Difference

Building this scraper required understanding Pinterest's technical landscape:

Pinterest Website Analyzer - Essential for understanding scraping challenges
Pinterest Scraping Guide - Comprehensive technical walkthrough
Scrapy Documentation - For framework fundamentals

Why I'm Sharing This (And the Code)

The Pinterest data opportunity is massive, but most developers are intimidated by the technical barriers. I spent weeks solving these challenges, and I believe the solution should be accessible.

I've open-sourced the complete Pinterest scraper on GitHub: pinterest-scrapy-scraper

The repository includes:

✅ Three specialized, production-ready spiders
✅ ScrapeOps proxy integration
✅ Complete documentation and usage examples
✅ CSV export pipeline

Getting Started (The Practical Way)

If you want to explore Pinterest data:

Clone the repository and review the implementation
Get a free ScrapeOps API key for proxy rotation (essential for success)
Start small - test with 5-10 items to understand the data structure
Scale gradually - respect rate limits and monitor success rates

The technical guide walks through the complete implementation process.

The Future of Visual Data Intelligence

Pinterest represents the future of visual search and content discovery. As AI and machine learning increasingly rely on visual data, platforms like Pinterest become crucial data sources for:

Computer vision training datasets
Consumer behavior analysis
Trend prediction algorithms
Content recommendation systems

Final Thoughts

Building a Pinterest scraper taught me that the most valuable data often sits behind the most challenging technical barriers. Pinterest's anti-bot measures exist for good reasons, but ethical, respectful scraping can coexist with platform protection.

The key is building tools that respect the platform while extracting meaningful insights. My Pinterest scraper does exactly that—and now it's available for anyone facing similar challenges.

Have you tackled Pinterest scraping challenges? Share your experiences in the comments. And if you build something interesting with the scraper, I'd love to hear about it!

Building a Production-Ready Target Scraper: 5 Spiders That Actually Work in 2024

Noorsimar Singh — Mon, 07 Jul 2025 17:03:45 +0000

How I built a comprehensive Target.com scraping solution and what I learned about e-commerce data extraction

Why I Built This (And Why You Might Need It Too)

Last month, I was helping a friend analyze pricing trends for their dropshipping business. They were manually checking Target prices for 200+ products every week – a soul-crushing task that was eating up 6+ hours of their time.

"There has to be a better way," I thought. That's when I decided to build a proper Target scraping solution.

After 3 weeks of development and testing, I ended up with 5 specialized spiders that can extract everything from product details to real-time deals. Here's what I learned along the way.

The Target Scraping Challenge: It's Harder Than You Think

Target isn't your average e-commerce site. They've got:

Advanced anti-bot detection (Cloudflare + behavioral analysis)
Dynamic JavaScript rendering for product data
Aggressive rate limiting (get blocked fast without proxies)
Complex page structures that change regularly
Dynamic URL identifiers that change per session

My first naive attempt? Blocked within 10 requests. 🤦‍♂️

But here's the thing – Target's data is incredibly valuable for:

Price monitoring and competitive analysis
Product research for dropshipping/affiliate marketing
Market trend analysis across categories
Inventory tracking for business intelligence

So I knew I had to crack this nut properly.

The 5-Spider Architecture That Actually Works

Instead of building one massive spider, I broke it down into specialized components:

1. target_search.py - Product Discovery

# Basic usage
scrapy crawl target_search -a search_query="laptop" -a max_products=50

Perfect for finding products by keywords and building product lists.

2. target_product.py - Deep Product Data

Extracts 48+ fields per product including:

Pricing (regular, sale, Target Circle offers)
Inventory status across fulfillment centers
Reviews and ratings
Product identifiers (TCIN, UPC, DPCI)

3. target_categories.py - Category Mapping

Discovers the entire Target category hierarchy – essential for systematic data collection.

4. target_deals.py - Promotions & Clearance

This one was tricky. Target's deals page is heavily JavaScript-dependent, but once I cracked it, it became a goldmine for finding discounted products.

5. target_hybrid.py - Dynamic Discovery

Combines category exploration with product extraction for comprehensive market analysis.

Technical Challenges & Solutions

Challenge 1: Dynamically Discovering Target's URL Identifiers

The Problem: Target doesn't use simple category slugs like /electronics or /home-garden. Instead, they use cryptic identifiers that look like this:

https://www.target.com/c/deals/-/N-atb3q
https://www.target.com/c/clearance/-/N-5q0ga
https://www.target.com/c/electronics/-/N-5xtg6

Those N-xxxxx codes change per user session and aren't static. Hardcoding them would be a nightmare.

The Smart Solution: Instead of manual discovery, I built automated identifier extraction directly into the spider:

def parse(self, response):
    """Extract category links and identifiers from Target's main pages"""

    # Parse navigation menu for category links
    category_links = response.css('nav a[href*="/c/"]::attr(href)').getall()

    # Extract deal section links
    deal_links = response.css('a[href*="deals"]::attr(href)').getall()

    # Parse embedded JSON for category mappings
    script_data = response.css('script:contains("navigation")::text').get()
    if script_data:
        nav_data = json.loads(script_data.split('=', 1)[1])
        categories = nav_data.get('categories', {})

        for category in categories:
            identifier = category.get('identifier')  # N-xxxxx code
            url = f"https://www.target.com/c/{category['slug']}/-/{identifier}"
            yield response.follow(url, self.parse_category)

Dynamic Discovery in Action:

def discover_deal_identifiers(self, response):
    """Automatically find current deal page identifiers"""

    # Target embeds category data in JavaScript
    category_script = response.css('script:contains("__TGT_DATA__")::text').get()

    if category_script:
        data = json.loads(category_script.split('=', 1)[1])

        # Extract deal categories with their current identifiers
        deal_categories = data.get('dealCategories', [])

        for deal in deal_categories:
            identifier = deal.get('navigationId')  # Current session identifier
            category_name = deal.get('displayName')

            deal_url = f"https://www.target.com/c/{deal['slug']}/-/{identifier}"

            yield {
                'category': category_name,
                'identifier': identifier,
                'url': deal_url
            }

Why This Approach Works:

Session-aware: Gets fresh identifiers for each scraping session
Scalable: Discovers new categories automatically
Maintenance-free: No hardcoded values that break over time
User-agnostic: Works regardless of user session or location

The Discovery Process:

# Step 1: Start from Target's homepage
start_urls = ['https://www.target.com/']

# Step 2: Extract navigation structure
def parse_homepage(self, response):
    # Get main navigation links
    nav_sections = response.css('[data-test="navigation-primary"] a')

    for section in nav_sections:
        category_url = section.css('::attr(href)').get()
        category_name = section.css('::text').get()

        if '/c/' in category_url:  # Category page
            yield response.follow(category_url, self.parse_category_page)

# Step 3: Extract identifiers from category pages
def parse_category_page(self, response):
    # Parse URL to extract identifier
    url_parts = response.url.split('/')
    identifier = url_parts[-1] if url_parts[-1].startswith('N-') else None

    if identifier:
        self.discovered_identifiers[category_name] = identifier

This approach means my spider automatically adapts to Target's changing URL structure without any manual intervention. The target_categories.py spider now discovers 12+ categories dynamically, and the target_deals.py spider finds all current deal types automatically.

Real Impact: This discovery system found deal categories I didn't even know existed, increasing the spider's effectiveness from 4-5 products to 39+ targeted deals per run.

Challenge 2: Getting Past Anti-Bot Detection

The Problem: Target blocks scrapers aggressively. Even with user agents and headers, I was getting 403s constantly.

The Solution: Proxy rotation was non-negotiable. After testing several services, I went with ScrapeOps because:

Free tier (1,000 requests) perfect for development
95%+ success rate with Target specifically
Built-in Scrapy integration (no complex setup)
Real-time monitoring dashboard

# Settings that actually work
SCRAPEOPS_API_KEY = 'your-api-key'  # Free from scrapeops.io
SCRAPEOPS_PROXY_ENABLED = True
CONCURRENT_REQUESTS = 1  # Start conservative
DOWNLOAD_DELAY = 1

Challenge 3: Handling Dynamic Content

Target loads product data via JavaScript. Static HTML scraping gets you maybe 30% of the actual data.

My approach: Targeted JSON extraction from embedded scripts rather than full browser automation. Faster and more reliable.

# Extract JSON data from script tags
json_data = response.css('script:contains("__TGT_DATA__")::text').get()
if json_data:
    # Parse the embedded product data
    product_info = json.loads(json_data.split('=', 1)[1])

Challenge 4: Field Mapping Hell

Target uses different data structures across pages. A product might have price in one place and pricing.current in another.

Solution: Created a robust TargetProductItem with 48+ fields and intelligent fallback extraction:

# Smart price extraction with fallbacks
item['price'] = (
    data.get('price', {}).get('current') or
    data.get('pricing', {}).get('current_retail') or
    response.css('.Price-module__price::text').get()
)

Results That Matter

After optimization, here's what the system delivers:

✅ 39 deals extracted per run from Target's deals page
✅ 95%+ success rate with proper proxy rotation
✅ 48+ data fields per product with 90%+ completion
✅ Clean CSV exports with timestamps
✅ Multiple deal types: BOGO, percentage discounts, clearance, Target Circle exclusives
✅ Automatic category discovery with dynamic identifier mapping

The friend I built this for? They've saved 6+ hours per week and discovered 23% more profitable products they were missing before.

Key Learnings for E-commerce Scraping

1. Build for Dynamic Discovery

Never hardcode identifiers or category URLs. E-commerce sites change these constantly. Build spiders that discover and adapt automatically.

2. Proxy Rotation is Essential

For any serious e-commerce scraping, especially Target. The free ScrapeOps tier handles development perfectly.

3. Monitor Success Rates

If you're below 90% success rate, something's wrong. Fix it before scaling.

4. Structure for Maintenance

E-commerce sites change layouts constantly. Build modular spiders that are easy to update.

5. Legal Compliance Matters

Always check robots.txt and Terms of Service. Respect the website's resources.

6. Start with Navigation Structure

Parse the main navigation first. It's your roadmap to everything else on the site.

The Complete Solution

I've open-sourced the entire scraper suite because I believe good tools should be shared. The codebase includes:

All 5 production-ready spiders with automated discovery
Complete setup automation (python setup.py)
Comprehensive documentation with real usage examples
Professional error handling and logging
Clean project structure ready for customization
Dynamic identifier discovery system that adapts to Target's changes

GitHub: The full source code is available with detailed setup instructions. Just search for "target-scrapy-scraper" – it includes everything you need to get started.

Quick Start: After cloning, you'll need a free ScrapeOps API key for proxy rotation. The setup script guides you through everything.

*How to Scrape Guide: * Here is the original target guide I followed.

*Website Analyzer: * This target scraping analyzer helped me with getting data types, understanding the difficulty, and legalities.

What's Next?

I'm considering adding:

Real-time price change alerts
Competitor price comparison across multiple retailers
Integration with Google Sheets/databases
Mobile app price data
Enhanced category discovery algorithms
Automated testing for identifier changes

If you end up using this scraper or building something similar, I'd love to hear about your experience. E-commerce data extraction is fascinating – there's always more to discover.

What e-commerce scraping challenges have you faced? Drop your experiences in the comments!

Why Google's Anti-Bot System Made Me a Better Developer (A Technical Journey)

Noorsimar Singh — Thu, 03 Jul 2025 14:00:00 +0000

The 403 Forbidden error that taught me everything about modern web scraping

2025-07-01 02:47:12 [scrapy.core.downloader] DEBUG: Retrying <GET https://www.google.com/search?q=python> (failed 3 times): 403 Forbidden

Staring at this error for the hundredth time, I realized I was approaching Google scraping all wrong. This isn't just another "how to scrape Google" tutorial – it's the story of how reverse-engineering Google's defenses taught me about browser fingerprinting, JavaScript parsing, and building truly resilient systems.

The Problem: Google Isn't Playing Fair (And That's Brilliant)

Every developer has been there. You write a beautiful Scrapy spider, test it on a few pages, deploy it confidently... and watch it fail spectacularly in production.

Google's anti-bot system is a masterclass in defensive engineering:

Dynamic CSS selectors that change between requests
JavaScript-encrypted data hidden in plain sight
Browser fingerprinting that makes basic User-Agent spoofing laughable
Rate limiting algorithms that adapt to scraping patterns

As frustrating as it is, I have to respect the engineering behind it.

The Technical Breakthrough: Understanding Browser Fingerprinting

The turning point came when I started analyzing real browser requests vs. my scraper's requests. The difference was shocking.

What I Was Sending (Amateur Hour):

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}

What Browsers Actually Send:

headers = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.9',
    'Accept-Encoding': 'gzip, deflate, br',
    'Cache-Control': 'max-age=0',
    'sec-ch-ua': '"Google Chrome";v="119", "Chromium";v="119", "Not?A_Brand";v="24"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"Windows"',
    'Sec-Fetch-Dest': 'document',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-Site': 'none',
    'Sec-Fetch-User': '?1',
    'Upgrade-Insecure-Requests': '1',
}

The sec-ch-ua and Sec-Fetch-* headers are security tokens that Chrome automatically includes. Without them, you're basically announcing "I'm a bot."

Deep Dive: Reverse Engineering Google Images

Here's where things got really interesting. While everyone focuses on HTML parsing, Google Images stores the real data in JavaScript objects.

The Decoy (What Everyone Tries):

# This gets you placeholder URLs - useless!
img_urls = response.css('img::attr(src)').getall()
# Result: ['data:image/gif;base64,R0lGODlhAQABAIAAAP...']

The Real Deal (What Actually Works):

import re

# Find JavaScript data containing real image URLs
scripts = response.xpath('//script/text()').getall()

for script in scripts:
    # Google's internal format: [1,[0,"id",["thumb_url",w,h],["full_url",w,h]]
    pattern = r'\[1,\[0,"[^"]+",\["([^"]+)",[0-9]+,[0-9]+\],\["([^"]+)",[0-9]+,[0-9]+\]'
    matches = re.findall(pattern, script)

    for thumbnail_url, full_image_url in matches:
        yield {
            'thumbnail_url': thumbnail_url,
            'image_url': full_image_url,  # This is the real high-res URL!
            'source_domain': urlparse(full_image_url).netloc
        }

This revelation changed everything. Instead of fighting Google's HTML, I was reading their internal data structures.

Building a Production-Ready Architecture

After countless iterations, I settled on a three-spider architecture that actually works:

# Project structure that scales
google_search_scraper/
├── spiders/
│   ├── google_search.py      # SERP results
│   ├── google_news.py        # News articles  
│   └── google_images.py      # Real image URLs
├── items.py                  # Data models
├── middlewares.py            # Custom logic
└── settings.py               # Configuration

Spider 1: Google Search (The Foundation)

def parse(self, response):
    # Multiple selectors for reliability
    search_results = response.css('div.tF2Cxc, div.g, div.Gx5Zad')

    for result in search_results:
        yield {
            'title': result.css('h3::text, .LC20lb::text').get(),
            'url': result.css('a::attr(href)').get(),
            'description': result.css('.VwiC3b::text, .s3v9rd::text').get(),
            'position': len(search_results) - len(search_results) + search_results.index(result) + 1
        }

Spider 2: Google News (The Challenge)

Google News was tricky because the selectors change frequently. My solution: adaptive parsing with fallbacks.

def parse_news(self, response):
    # Primary selector
    news_containers = response.css('div.SoaBEf')

    if not news_containers:
        # Fallback selectors for different layouts
        news_containers = response.css('div.Gx5Zad, div.g')

    for container in news_containers:
        item = GoogleNewsItem()
        item['title'] = container.css('div.MBeuO::text, h3::text').get()
        item['url'] = container.css('a::attr(href)').get()
        item['source'] = container.css('div.MgUUmf span::text').get()
        item['date'] = container.css('div.LfVVr::text, span.r0bn4c::text').get()

        if item['url'] and item['title']:
            yield item

Spider 3: Google Images (The Innovation)

This is where the JavaScript parsing really shines:

def extract_image_data(self, response):
    all_scripts = response.xpath('//script/text()').getall()

    for script in all_scripts:
        if any(pattern in script for pattern in ['BNrT', 'encrypted-tbn', 'https://']):
            # Extract real URLs from Google's data structure
            pattern1 = r'"BNrT[a-zA-Z0-9]{2}":\s*\[1,\[0,"[^"]+",\["([^"]+)",[0-9]+,[0-9]+\],\["([^"]+)",[0-9]+,[0-9]+\]'
            matches = re.findall(pattern1, script)

            for thumbnail_url, full_image_url in matches:
                yield GoogleImageItem(
                    image_url=full_image_url,
                    thumbnail_url=thumbnail_url,
                    source_domain=urlparse(full_image_url).netloc
                )

The Infrastructure Problem: Why DIY Proxies Don't Work

Here's the uncomfortable truth: You can't reliably scrape Google with basic proxies.

I tried everything:

Residential proxy services (inconsistent)
VPN rotation (too slow)
Cloud server hopping (gets detected fast)
Free proxy lists (complete waste of time)

What finally worked was understanding that Google scraping requires specialized infrastructure that understands Google's patterns. After researching various solutions, I found that proxy aggregation services designed specifically for scraping perform significantly better than DIY approaches.

The key insight: Don't compete with Google's infrastructure; use professional tools that already solved this problem.

Real Performance Numbers (That Actually Matter)

After implementing these techniques, here's what I achieved:

# Google Search Results
2025-07-01 11:33:45 [scrapy.core.engine] INFO: Spider closed (finished)
{'item_scraped_count': 55, 'request_count': 6, 'response_status_count/200': 6}

# Google News Articles  
2025-07-01 11:35:12 [scrapy.core.engine] INFO: Spider closed (finished)
{'item_scraped_count': 56, 'request_count': 6, 'response_status_count/200': 6}

# Google Images (Real URLs!)
2025-07-01 11:39:43 [scrapy.core.engine] INFO: Spider closed (finished)
{'item_scraped_count': 300, 'items_per_minute': 782.6, 'response_status_count/200': 6}

300 real image URLs instead of placeholder data URLs. That's the difference between my amateur and professional scraping infrastructure.

The Development Setup That Actually Works

Prerequisites:

# Python 3.8+ required
python --version

# Create isolated environment
python -m venv scraper_env
source scraper_env/bin/activate  # Linux/Mac
# scraper_env\Scripts\activate  # Windows

# Install dependencies
pip install scrapy scrapeops-scrapy scrapeops-scrapy-proxy-sdk

Configuration:

# settings.py - The magic happens here
BOT_NAME = 'google_search_scraper'
ROBOTSTXT_OBEY = False  # Google's robots.txt blocks everything

# Professional-grade headers
DEFAULT_REQUEST_HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.9',
    'sec-ch-ua': '"Google Chrome";v="119", "Chromium";v="119", "Not?A_Brand";v="24"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"Windows"',
}

# Smart throttling
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1
AUTOTHROTTLE_MAX_DELAY = 10
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0

# This is where the real magic happens - professional proxy integration
SCRAPEOPS_API_KEY = 'your-api-key'  # Free tier: 1000 requests
SCRAPEOPS_PROXY_ENABLED = True

DOWNLOADER_MIDDLEWARES = {
    'scrapeops_scrapy_proxy_sdk.scrapeops_scrapy_proxy_sdk.ScrapeOpsScrapyProxySdk': 725,
}

Testing Your Setup

# Start with a simple test
scrapy crawl google_search

# Check the data quality
head -n 5 data/google_search_*.csv

# Try the advanced features
scrapy crawl google_images

If you see real image URLs instead of data:image/gif placeholders, you know it's working.

Advanced Debugging Techniques

Monitor Your Success Rate:

# Add this to your spider for real-time monitoring
def closed(self, reason):
    stats = self.crawler.stats
    self.logger.info(f"Scraped {stats.get_value('item_scraped_count')} items")
    self.logger.info(f"Success rate: {stats.get_value('response_status_count/200', 0) / stats.get_value('request_count', 1) * 100:.1f}%")

CSS Selector Debugging:

# Use Scrapy shell for live testing
scrapy shell "https://www.google.com/search?q=python"

# Test your selectors interactively
>>> response.css('div.tF2Cxc').get()

Proxy Performance Analysis:

# Monitor proxy performance in your spider
def parse(self, response):
    proxy_used = response.meta.get('proxy')
    self.logger.info(f"Response from proxy: {proxy_used}")

What I Learned About Modern Web Scraping

Building this scraper taught me several important lessons:

Browser fingerprinting is the new frontier - User-Agent headers are just the beginning
JavaScript parsing often beats HTML parsing - The real data might not be in the DOM
Professional infrastructure matters - Some problems are worth paying to solve
Adaptive selectors are essential - Websites change; your scrapers must adapt
Monitoring is crucial - You need real-time feedback on what's working

The Open Source Contribution

I've made the complete scraper available as an open-source project: google-search-scrapy-scraper

What's included:

All three production-ready spiders
Comprehensive configuration examples
Debugging tools and monitoring setup
Regular updates as Google's systems evolve

Why open source? Because the scraping community helped me learn, and I want to give back. Plus, Google's defenses evolve constantly – community collaboration makes us all more effective.

Performance Optimization Tips

1. Batch Your Requests

def start_requests(self):
    keywords = ['python', 'javascript', 'rust']  # Batch related keywords
    for keyword in keywords:
        for page in range(0, 2):  # First 2 pages only
            yield Request(url=f"https://www.google.com/search?q={keyword}&start={page*10}")

2. Smart Error Handling

def parse(self, response):
    if response.status != 200:
        self.logger.warning(f"Non-200 response: {response.status}")
        return

    results = response.css('div.tF2Cxc')
    if not results:
        self.logger.warning("No results found - possible selector change")
        # Trigger alert or fallback logic

3. Data Validation

def parse_item(self, response):
    item = GoogleSearchItem()
    item['url'] = response.css('a::attr(href)').get()

    # Validate before yielding
    if item['url'] and item['url'].startswith('http'):
        yield item
    else:
        self.logger.warning(f"Invalid URL: {item['url']}")

The Future of Google Scraping

Based on my analysis, here's where I think things are heading:

AI-powered anti-bot systems will become more sophisticated
JavaScript-heavy interfaces will replace static HTML
Legal frameworks will become more defined
Professional tooling will become essential (DIY approaches will fail more often)

Getting Started: Your Action Plan

Clone the repository and explore the code structure
Set up your development environment with proper dependencies
Get a free API key for testing (1000 requests should be plenty to start)
Run the spiders and analyze the output quality
Customize for your use case - modify keywords, selectors, output formats
Monitor and iterate - Google changes, so should your scrapers

Final Thoughts: Why This Matters

Google scraping isn't just about extracting data – it's about understanding modern web architecture, defensive programming, and building resilient systems.

The techniques I've shared here apply far beyond Google. Browser fingerprinting, JavaScript parsing, and professional infrastructure management are skills that make you a better developer overall.

Try the scraper. Break it. Improve it. Share your findings with the community. That's how we all get better.

Resources for Going Deeper

Complete scraper code - Production-ready implementation
Google scraping analysis - Technical deep-dive into Google's defenses
Advanced techniques guide - Next-level strategies

What's your experience with scraping challenges? Drop a comment – I'm always interested in hearing about novel approaches and war stories from the trenches.

Building a Production-Ready LinkedIn Scraper with Python Scrapy 🐍

Noorsimar Singh — Wed, 02 Jul 2025 07:47:07 +0000

A complete guide to extracting job data, company profiles, and professional insights at scale

TL;DR

I built a comprehensive LinkedIn scraper using Python Scrapy that can extract:

Job listings with pagination (175+ jobs extracted in testing)
Company profiles with business intelligence
Professional profiles with experience data
Anti-bot protection bypass with proxy rotation
Structured JSON output with automatic validation

🔗 Full source code on GitHub

The Problem

LinkedIn's API is severely limited - you can only access your own data and connected profiles. For comprehensive data extraction (job market analysis, recruitment intelligence, competitive research), web scraping becomes essential.

But LinkedIn implements aggressive anti-scraping measures:

Sophisticated bot detection
Rate limiting and IP blocking
JavaScript-heavy dynamic content
CAPTCHA challenges for suspicious activity

The Solution: Professional Scrapy Architecture

Here's the scraper architecture I built:

linkedin-scrapy-scraper/
├── linkedin/
│   ├── spiders/
│   │   ├── linkedin_jobs.py              # Jobs scraper (✅ Working)
│   │   ├── linkedin_company_profile.py   # Company data extractor
│   │   └── linkedin_people_profile.py    # Profile harvester
│   ├── middlewares.py                     # Anti-detection middleware
│   ├── items.py                          # Data models
│   ├── pipelines.py                      # Data processing
│   └── settings.py                       # ScrapeOps integration
├── data/                                 # Scraped data output
└── .gitignore                           # Clean repo management

Quick Start

# Clone and setup
git clone https://github.com/Simple-Python-Scrapy-Scrapers/linkedin-scrapy-scraper.git
cd linkedin-scrapy-scraper
python -m venv .venv && .venv\Scripts\activate
pip install scrapy scrapeops-scrapy scrapeops-scrapy-proxy-sdk

# Run the job scraper
python -m scrapy crawl linkedin_jobs

Deep Dive: Jobs Spider Implementation

The jobs spider is the most reliable since it uses LinkedIn's public job search endpoints:

import scrapy
from urllib.parse import urlencode

class LinkedinJobsSpider(scrapy.Spider):
    name = 'linkedin_jobs'

    # Auto-save to timestamped JSON Lines
    custom_settings = {
        'FEEDS': {
            'data/%(name)s_%(time)s.jsonl': {'format': 'jsonlines'}
        }
    }

    def start_requests(self):
        # Target multiple job categories
        queries = [
            'python developer', 'data scientist', 'devops engineer',
            'frontend developer', 'backend developer', 'full stack'
        ]

        for query in queries:
            params = {
                'keywords': query,
                'location': 'United States',
                'geoId': '103644278',
                'start': 0
            }

            url = 'https://www.linkedin.com/jobs-guest/jobs/api/seeMoreJobPostings/search?' + urlencode(params)

            yield scrapy.Request(
                url=url,
                callback=self.parse_jobs,
                meta={'query': query, 'page': 0}
            )

    def parse_jobs(self, response):
        jobs = response.css('.result-card')

        for job in jobs:
            # Extract comprehensive job data
            yield {
                'job_title': job.css('h3.result-card__title a::text').get(),
                'company_name': job.css('h4.result-card__subtitle a::text').get(),
                'company_location': job.css('.job-result-card__location::text').get(),
                'job_listed': job.css('time.job-result-card__listdate::attr(datetime)').get(),
                'job_detail_url': job.css('h3.result-card__title a::attr(href)').get(),
                'company_link': job.css('h4.result-card__subtitle a::attr(href)').get(),
                'query': response.meta['query'],
                'scraped_at': datetime.now().isoformat()
            }

        # Smart pagination with limits
        if jobs and response.meta['page'] < 10:
            next_page = response.meta['page'] + 1
            params = {
                'keywords': response.meta['query'],
                'location': 'United States',
                'geoId': '103644278',
                'start': next_page * 25
            }

            next_url = 'https://www.linkedin.com/jobs-guest/jobs/api/seeMoreJobPostings/search?' + urlencode(params)

            yield scrapy.Request(
                url=next_url,
                callback=self.parse_jobs,
                meta={
                    'query': response.meta['query'], 
                    'page': next_page
                }
            )

Anti-Detection Strategies

1. Proxy Rotation with ScrapeOps

LinkedIn blocks IPs aggressively. ScrapeOps provides residential proxy rotation:

# settings.py
SCRAPEOPS_API_KEY = 'your_free_api_key'  # Get at scrapeops.io
SCRAPEOPS_PROXY_ENABLED = True

DOWNLOADER_MIDDLEWARES = {
    'scrapeops_scrapy_proxy_sdk.scrapeops_scrapy_proxy_sdk.ScrapeOpsScrapyProxySdk': 725,
}

# Conservative rate limiting
CONCURRENT_REQUESTS = 1
DOWNLOAD_DELAY = 2
RANDOMIZE_DOWNLOAD_DELAY = 0.5
AUTOTHROTTLE_ENABLED = True

2. User Agent Rotation Middleware

# middlewares.py
import random
from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware

class RotateUserAgentMiddleware(UserAgentMiddleware):
    def __init__(self):
        self.user_agent_list = [
            'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
            'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
        ]

    def process_request(self, request, spider):
        ua = random.choice(self.user_agent_list)
        request.headers['User-Agent'] = ua
        return None

3. Advanced Error Handling

# Custom retry middleware for LinkedIn-specific errors
class LinkedInRetryMiddleware:
    def process_response(self, request, response, spider):
        if response.status == 999:  # LinkedIn's anti-bot response
            spider.logger.warning(f"LinkedIn 999 error for {request.url}")
            return self._retry(request, spider)

        if "challenge" in response.url:  # CAPTCHA redirect
            spider.logger.warning(f"CAPTCHA challenge detected for {request.url}")
            return self._retry(request, spider)

        return response

    def _retry(self, request, spider):
        retries = request.meta.get('retry_times', 0) + 1
        if retries <= 3:
            retry_req = request.copy()
            retry_req.meta['retry_times'] = retries
            return retry_req
        return None

Company Profile Spider

Extract business intelligence from LinkedIn company pages:

class LinkedinCompanySpider(scrapy.Spider):
    name = 'linkedin_company_profile'

    def parse_company(self, response):
        # Extract comprehensive company data
        company_data = {
            'name': response.css('h1.org-top-card-summary__title::text').get(),
            'industry': response.css('.org-top-card-summary__industry::text').get(),
            'company_size': response.css('.org-about-company-module__company-size-definition-text::text').get(),
            'founded_year': response.css('.org-about-company-module__founded span::text').get(),
            'headquarters': response.css('.org-about-company-module__headquarters span::text').get(),
            'description': response.css('.org-about-company-module__description::text').get(),
            'website': response.css('.org-about-company-module__website a::attr(href)').get(),
            'employee_count': response.css('.org-about-company-module__company-staff-count-range::text').get(),
            'follower_count': response.css('.org-top-card-summary__follower-count::text').get(),
        }

        # Extract specialties/keywords
        specialties = response.css('.org-about-company-module__specialties dd::text').getall()
        company_data['specialties'] = [spec.strip() for spec in specialties if spec.strip()]

        # Extract recent posts/updates
        updates = []
        for update in response.css('.org-update'):
            updates.append({
                'title': update.css('.org-update__title::text').get(),
                'timestamp': update.css('.org-update__time::text').get(),
                'content': update.css('.org-update__content::text').get()
            })
        company_data['recent_updates'] = updates

        yield company_data

Professional Profile Spider

Extract detailed professional information:

class LinkedinPeopleSpider(scrapy.Spider):
    name = 'linkedin_people_profile'

    def parse_profile(self, response):
        # Basic profile info
        profile = {
            'name': response.css('.text-heading-xlarge::text').get(),
            'headline': response.css('.text-body-medium.break-words::text').get(),
            'location': response.css('.text-body-small.inline.t-black--light::text').get(),
            'connections': response.css('.t-black--light .t-bold::text').get(),
            'about': response.css('.pv-about-section .pv-about__summary-text::text').get()
        }

        # Extract experience
        experience = []
        for exp in response.css('.pv-profile-section.experience .pv-entity__position-group'):
            exp_data = {
                'title': exp.css('.pv-entity__summary-info h3::text').get(),
                'company': exp.css('.pv-entity__secondary-title::text').get(),
                'location': exp.css('.pv-entity__location span::text').get(),
                'duration': exp.css('.pv-entity__date-range span::text').get(),
                'description': exp.css('.pv-entity__description::text').get()
            }
            experience.append(exp_data)

        profile['experience'] = experience

        # Extract education
        education = []
        for edu in response.css('.pv-profile-section.education .pv-entity__position-group'):
            edu_data = {
                'school': edu.css('.pv-entity__school-name::text').get(),
                'degree': edu.css('.pv-entity__degree-name span::text').get(),
                'field_of_study': edu.css('.pv-entity__fos span::text').get(),
                'dates': edu.css('.pv-entity__dates span::text').get()
            }
            education.append(edu_data)

        profile['education'] = education

        # Extract skills
        skills = response.css('.pv-skill-category-entity__name span::text').getall()
        profile['skills'] = [skill.strip() for skill in skills if skill.strip()]

        yield profile

Data Pipeline & Validation

# pipelines.py
import json
from itemadapter import ItemAdapter
from scrapy.exceptions import DropItem

class ValidationPipeline:
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)

        # Validate required fields based on spider type
        if spider.name == 'linkedin_jobs':
            if not adapter.get('job_title') or not adapter.get('company_name'):
                raise DropItem(f"Missing required fields in {item}")

        elif spider.name == 'linkedin_company_profile':
            if not adapter.get('name'):
                raise DropItem(f"Missing company name in {item}")

        elif spider.name == 'linkedin_people_profile':
            if not adapter.get('name'):
                raise DropItem(f"Missing profile name in {item}")

        return item

class DataCleaningPipeline:
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)

        # Clean and normalize text fields
        for field_name, field_value in adapter.items():
            if isinstance(field_value, str):
                # Remove extra whitespace and newlines
                cleaned_value = ' '.join(field_value.split())
                adapter[field_name] = cleaned_value

        return item

class JsonExportPipeline:
    def open_spider(self, spider):
        self.file = open(f'data/{spider.name}_detailed.json', 'w')
        self.file.write('[\n')
        self.first_item = True

    def close_spider(self, spider):
        self.file.write('\n]')
        self.file.close()

    def process_item(self, item, spider):
        if not self.first_item:
            self.file.write(',\n')
        else:
            self.first_item = False

        line = json.dumps(ItemAdapter(item).asdict(), indent=2)
        self.file.write(line)
        return item

Performance Metrics & Testing

In my testing environment:

# Jobs Spider Results
✅ 175+ jobs extracted across 7+ pages
✅ 68KB+ structured data per session  
✅ 100% field extraction success rate
✅ Zero errors with proper rate limiting
✅ Average 1.2 seconds per job with delays

# File output structure
data/
├── linkedin_jobs_2024-01-15_14-30-25.jsonl     # 68KB
├── linkedin_company_profile_2024-01-15.jsonl   # 45KB  
└── linkedin_people_profile_2024-01-15.jsonl    # 112KB

Scaling for Production

ScrapeOps Integration

ScrapeOps provides enterprise proxy infrastructure:

# Free tier: 1,000 requests
# Perfect for development and testing
pip install scrapeops-scrapy-proxy-sdk

# Production settings
SCRAPEOPS_API_KEY = 'your_free_api_key'  
SCRAPEOPS_PROXY_ENABLED = True
SCRAPEOPS_PROXY_SETTINGS = {
    'country': 'us',
    'render_js': False,
    'residential': True
}

Monitoring & Analytics

# Enable ScrapeOps monitoring
EXTENSIONS = {
    'scrapeops_scrapy.extension.ScrapeOpsMonitor': 500,
}

# Real-time scraping metrics:
# - Success/failure rates
# - Response times  
# - Proxy performance
# - Error categorization

Common Issues & Solutions

HTTP 999 Errors

# Solution: Enable residential proxies
SCRAPEOPS_PROXY_SETTINGS = {'residential': True}

JavaScript Content Loading

# Solution: Use Scrapy-Splash
pip install scrapy-splash
# Or enable JS rendering in ScrapeOps
SCRAPEOPS_PROXY_SETTINGS = {'render_js': True}

Rate Limiting

# Conservative approach for LinkedIn
DOWNLOAD_DELAY = 3
RANDOMIZE_DOWNLOAD_DELAY = 0.5
CONCURRENT_REQUESTS = 1

Real-World Applications

This scraper has been used for:

Job Market Analysis 📊

# Analyze salary trends by location/technology
jobs_df = pd.read_json('data/linkedin_jobs.jsonl', lines=True)
salary_trends = jobs_df.groupby(['location', 'technology']).agg({
    'salary': 'mean',
    'job_title': 'count'
}).reset_index()

Recruitment Intelligence 👥

# Track competitor hiring patterns
company_jobs = jobs_df[jobs_df['company_name'].isin(competitors)]
hiring_velocity = company_jobs.groupby('company_name').size()

Lead Generation 🎯

# Identify growing companies in your sector
growing_companies = companies_df[
    (companies_df['employee_count_change'] > 20) & 
    (companies_df['industry'] == 'Software')
]

Security & Legal Considerations

# Implement respectful scraping
ROBOTSTXT_OBEY = True  # Respect robots.txt
DOWNLOAD_DELAY = 2     # Don't overwhelm servers

# Data privacy compliance
class PrivacyPipeline:
    def process_item(self, item, spider):
        # Remove PII for GDPR compliance
        if 'email' in item:
            del item['email']
        if 'phone' in item:
            del item['phone']
        return item

Getting Started

Clone the repo:

git clone https://github.com/Simple-Python-Scrapy-Scrapers/linkedin-scrapy-scraper.git

Get free ScrapeOps API key: scrapeops.io/app/register/main
Run your first scrape:

python -m scrapy crawl linkedin_jobs

Analyze the data:

import pandas as pd
df = pd.read_json('data/linkedin_jobs_*.jsonl', lines=True)
print(df.describe())

What's Next?

🔄 Real-time monitoring with job alerts
🤖 ML integration for salary prediction
📊 Dashboard creation with Streamlit/Dash
🌐 Multi-region support with geo-targeted proxies
📈 Advanced analytics with trend detection

Resources

Found this helpful? ⭐ Star the repository and follow for more web scraping tutorials!

Questions? Drop them in the comments below 👇

Want to collaborate? Open an issue or submit a PR!

From 500+ Lines to 150: How I Refactored My Amazon Scraper Into a Clean Scrapy Project

Noorsimar Singh — Wed, 25 Jun 2025 15:22:09 +0000

The Messy Beginning

Like many developers, I started Amazon scraping with a single Python file. What began as a "quick script" quickly turned into 500+ lines of spaghetti code:

# The old way - amazon_spider_old.py (don't do this!)
import requests
from bs4 import BeautifulSoup
import csv
import time
import random
from fake_useragent import UserAgent

# 50+ lines of proxy management
# 100+ lines of parsing logic
# 200+ lines of error handling
# 150+ lines of data export
# Complete nightmare to maintain!

Sound familiar? Here's how I transformed this mess into a clean, maintainable Scrapy project.

The Pain Points of DIY Scraping

Problem #1: IP Blocks Everywhere

# My hacky solution (don't judge!)
proxies = [
    "http://proxy1:8000",
    "http://proxy2:8000", 
    # ... manually managing 20+ proxies
]

for i in range(len(proxies)):
    try:
        response = requests.get(url, proxies={"http": proxies[i]})
        if response.status_code == 200:
            break
    except:
        continue  # Prayer-driven development 🙏

Problem #2: Fragile Selectors

# This broke every week
try:
    price = soup.select('.a-price-whole')[0].text
except:
    try:
        price = soup.select('.a-offscreen')[0].text
    except:
        try:
            price = soup.select('.a-price')[0].text
        except:
            price = "Not found"  # 😭

Problem #3: Zero Monitoring

When scraping failed (and it failed often), I had no idea why:

Was it the proxy?
Did Amazon change their layout?
Network issues?
Rate limiting?

Debugging was pure guesswork.

The Scrapy Transformation

After one too many 3 AM debugging sessions, I decided to rebuild using Scrapy. Here's the before/after:

Before: Single Monolithic File

amazon_spider_old.py (534 lines)
├── Proxy management (manual) 
├── User agent rotation (manual)
├── Request retries (manual)
├── Data parsing (fragile)
├── CSV export (buggy)
└── Error handling (prayer-based)

After: Clean Scrapy Architecture

amazon-scrapy-scraper/
├── amazon_scraper/
│   ├── spiders/
│   │   ├── amazon_search.py      (72 lines)
│   │   └── amazon_product.py     (78 lines)
│   ├── settings.py               (31 lines)
│   └── data/                     (auto-generated)
├── requirements.txt              (3 lines)
└── README.md                     (comprehensive docs)

Result: 500+ lines → 150 lines total! 🎉

Key Architectural Decisions

1. Single Responsibility Spiders

Instead of one spider doing everything, I created focused spiders:

Search Spider (amazon_search.py):

class AmazonSearchSpider(scrapy.Spider):
    name = 'amazon_search'

    def start_requests(self):
        keyword_list = ['ipad']
        for keyword in keyword_list:
            url = f'https://www.amazon.com/s?k={keyword}'
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        # Extract search results
        products = response.css('div[data-component-type="s-search-result"]')
        for product in products:
            yield {
                'title': product.css('h2 a span::text').get(),
                'price': product.css('.a-price-whole::text').get(),
                'url': product.css('h2 a::attr(href)').get(),
            }

Product Spider (amazon_product.py):

class AmazonProductSpider(scrapy.Spider):
    name = 'amazon_product'

    def parse_product_data(self, response):
        # Detailed product extraction
        yield {
            "name": response.css("#productTitle::text").get("").strip(),
            "price": self.extract_price(response),
            "features": self.extract_features(response),
            "images": self.extract_images(response),
        }

2. Configuration Over Code

All settings in one place:

# settings.py
BOT_NAME = 'amazon_scraper'
ROBOTSTXT_OBEY = False
CONCURRENT_REQUESTS = 1

# ScrapeOps Integration (game changer!)
SCRAPEOPS_API_KEY = 'your-api-key'
SCRAPEOPS_PROXY_ENABLED = True

DOWNLOADER_MIDDLEWARES = {
    'scrapeops_scrapy_proxy_sdk.scrapeops_scrapy_proxy_sdk.ScrapeOpsScrapyProxySdk': 725,
}

# Auto-export to CSV
FEEDS = {
    'data/%(name)s_%(time)s.csv': {'format': 'csv'}
}

3. Built-in Superpowers

Scrapy + ScrapeOps gives you so much for free:

Scrapy provides:

Automatic retries with exponential backoff
Concurrent requests with rate limiting
Robust selectors with fallbacks
Multiple export formats (CSV, JSON, XML)
Middleware system for custom logic

ScrapeOps provides:

Proxy rotation and IP management
Real-time monitoring dashboard
Request success tracking

The ScrapeOps Game Changer

The biggest improvement wasn't Scrapy itself—it was integrating ScrapeOps:

Before: Manual Proxy Hell

# 50+ lines of proxy management code
proxies = load_proxy_list()
current_proxy = 0

def get_next_proxy():
    global current_proxy
    if current_proxy >= len(proxies):
        current_proxy = 0
        time.sleep(60)  # Wait and pray
    proxy = proxies[current_proxy]
    current_proxy += 1
    return proxy

After: One Line Integration

# settings.py
SCRAPEOPS_PROXY_ENABLED = True

That's it! ScrapeOps handles:

Proxy rotation
IP geolocation
User agent rotation
Request success monitoring
Automatic retries

Monitoring Dashboard

The ScrapeOps dashboard shows real-time metrics:

Request success rates
Response times
Error patterns
Bandwidth usage

So.. No more guessing what went wrong.

Performance Comparison

Metric	Old Script	New Scrapy Project
Lines of Code	534	150
Success Rate	~70%	~95%
Maintainability	Nightmare	Easy
Debugging Time	Hours	Minutes
Scalability	Limited	Unlimited
Monitoring	None	Full dashboard

Real-World Usage Examples

Quick Product Search

# Search for iPads
scrapy crawl amazon_search

# Output: data/amazon_search_2024-01-15T10-30-45.csv

Detailed Product Analysis

# Get comprehensive product data
scrapy crawl amazon_product

# Output includes features, images, variants

Custom Keywords

# Edit amazon_search.py
keyword_list = ['mechanical keyboards', 'gaming mice', 'monitors']

Lessons for Fellow Developers

1. Don't Reinvent the Wheel

I wasted months building retry logic and export functionality that Scrapy provides out of the box, plus proxy rotation that ScrapeOps handles.

2. Monitoring is Essential

Without ScrapeOps monitoring, I was flying blind. Now I can optimize based on real data.

3. Architecture Matters

Two focused spiders are infinitely more maintainable than one giant spider.

4. Start with Free Tiers

ScrapeOps free tier (1,000 requests/month) is perfect for development and small projects.

Getting Started (5 Minutes Setup)

Clone and setup:

   git clone https://github.com/Simple-Python-Scrapy-Scrapers/amazon-scrapy-scraper.git
   cd amazon-scrapy-scraper
   python -m venv .venv
   source .venv/bin/activate  # or .venv\Scripts\Activate.ps1 on Windows
   pip install -r requirements.txt

Get free ScrapeOps API key:
- Visit ScrapeOps.io
- 1,000 free requests/month
Configure and run:

   # Add API key to amazon_scraper/settings.py
   cd amazon_scraper
   scrapy crawl amazon_search

What's Next?

This architecture scales beautifully. I'm already planning:

Review scraper for sentiment analysis
Price tracking with alerts
Multi-marketplace support (eBay, Walmart)
API endpoints for real-time data

The clean foundation makes adding features straightforward.

Repository & Resources

🔗 Full source code: amazon-scrapy-scraper

📚 Helpful resources:

Discussion

Have you refactored similar projects? What architectural patterns work best for web scraping?

Drop a comment with:

Your biggest scraping pain points
Scrapy tips and tricks
Alternative tools you've tried

Let's help each other build better scrapers!

P.S. - Web scraping should always respect robots.txt and rate limits. This project is for educational purposes.