DEV Community: readysetscrape

Ever wondered what problems a web scraping company runs into? I shared some lessons from my own experience in this post :) https://dev.to/readysetscrape/what-problems-do-we-face-as-a-web-scraping-company-5i6

readysetscrape — Thu, 17 Sep 2026 08:03:53 +0000

readysetscrape

Sep 16

What problems do we face as a Web Scraping Company?

#showdev #programming #programmers #python

5 min read

What problems do we face as a Web Scraping Company?

readysetscrape — Wed, 16 Sep 2026 13:08:48 +0000

Web scraping companies have to keep data flowing while websites block requests, proxies fail, servers go down, and deployments break working scrapers. Downloading HTML and extracting data is usually the easy part. Reliable delivery is the difficult part.

What makes running a web scraping company difficult?

Running a web scraping company means dealing with two kinds of problems at once: collecting the data and keeping the service alive. Clients see an API response, a CSV file, or fresh records in their system. They don't see everything required to make that delivery happen every day.

In our work, the recurring problems are:

CAPTCHAs, browser fingerprint checks, rate limits, and blocked requests.
Servers and critical processes that fail.
Proxy outages, poor proxy quality, and exhausted balances.
Deployments that damage shared infrastructure.
Failures that happen overnight or during a holiday.
Prospects who take a custom sample and then disappear.

Getting the data once proves very little. Keeping it flowing is the job.

Why isn't generic web scraping advice enough?

Generic web scraping advice names useful tactics, but it does not tell you how to build a reliable service for a specific website. The missing details are the ones that take years of testing, mistakes, blocks, and rebuilds to learn.

You can find plenty of advice online:

Use real browsers.
Keep cookies when sending HTTP requests.
Rotate proxy IPs.
Use residential proxies.
Make your traffic look more natural.

None of that is wrong. It is also nowhere near a complete solution.

Every scraping company has its own methods for dealing with blocks and detection systems. Companies that make money from those methods rarely publish the exact details. Those tools and operating methods are intellectual property.

Would you tell everyone where you dig for gold?

I doubt it.

Why is recurring data delivery harder than scraping once?

A successful scrape shows that the collector worked at one moment. Recurring delivery must also work tomorrow, next weekend, and on Christmas morning.

One or two servers do not give you continuity. Many things can interrupt a working service:

A hosting provider shuts down a server after noticing the scraping activity. Yes, this happens.
A scraper sends traffic without the intended proxy and causes problems for the provider.
Someone forgets to pay for a server. This also happens. We are only human.
A whole data center has an outage.
A server stays online while one critical process dies.

Having a server is not infrastructure. It is only the beginning.

What happens when a proxy service fails?

When a proxy service fails, requests can stop even if the scraper and its servers are healthy. A proxy is another dependency, and every dependency can fail.

In practice, we have to plan for several problems:

The proxy provider has a technical failure.
A faulty scraper burns through the available balance.
Someone forgets to pay the invoice.
A low-balance or expiry alert never reaches us.
Proxy quality changes overnight, so requests that worked yesterday are suddenly blocked.

The client bought the service from us. They should not have to care whether our server provider had an outage or our proxy network failed.

In their eyes, we are responsible.

That is fair...?

How can one bad deployment break other scraping jobs?

One bad deployment can consume shared resources and disrupt jobs that were working perfectly well. Even a small scraper change can damage the rest of the infrastructure.

A faulty scraper can:

Drain the proxy balance in minutes.
Take all available CPU and memory from a machine.
Flood a database with writes.
Fill a message queue or create millions of duplicate jobs.
Break a downstream data aggregation process.

A production scraper is rarely one script running quietly in the background. It may depend on databases, message queues, schedulers, storage, APIs, aggregation systems, monitoring, and alerts.

That means a deployment needs a safe way to stop a dangerous job and restore the last working version. Otherwise, one mistake can spread far beyond one scraper.

What should web scraping monitoring detect?

Web scraping monitoring should detect whether the expected data was collected and delivered, not merely whether a server responds. A green server status tells you very little when a critical process is dead.

Useful monitoring needs to catch problems such as:

A scheduled job that did not start or finish.
A sudden drop in successful requests or collected records.
More errors, retries, or blocked requests than usual.
Unexpected proxy usage or server load.
A growing queue or delayed delivery.

Depending on the failure, the system may restart a process, stop a dangerous job, switch to another server, disable a broken scraper, or alert the right person. Automated recovery also needs limits. Endless retries can turn one failure into a much more expensive failure.

The worst incidents happen while you sleep. Or during a holiday, exactly when you finally want to rest.

Scraping has a special sense of humor.

How does downtime affect client relationships?

Repeated downtime costs trust and can eventually cost the client. One short interruption is not the same as a pattern of unreliable delivery, but patience runs out when the same failures keep returning.

The client needs to know what happened, which data was affected, and whether missed data can be recovered. For recurring work, both sides should agree on the refresh schedule, what counts as a completed run, and how missed runs will be handled.

Clients rarely see this part of the job. They see the output. They do not see the servers, failed jobs, alerts, repairs, and backup plans behind it.

Why are data samples and payments a business risk?

Custom data samples cost time and infrastructure even when the prospect never becomes a client. We have prepared work for people who promised to pay for a sample and then did not pay.

Before receiving the sample, some prospects answer within minutes. After receiving it, every reply suddenly takes a week, if it comes at all.

Nothing physically stops a prospect from taking a sample and disappearing. That is why free evaluation data and paid custom work need a clear boundary. Before starting, agree on:

The target website and requested fields.
The sample size and delivery format.
Whether the sample is free or paid.
The price and payment timing for paid work.
What happens after the prospect reviews the sample.

We spend time and money doing this work. The person on the other side may simply not care.

That is another part of the scraping business nobody likes to mention.

So, is web scraping easy?

Web scraping looks simple because the visible part can be reduced to two steps:

Download the HTML.
Extract the data.

Everything else determines whether the client receives useful data tomorrow.

Easy, right?

Don't make me laugh.

If you have any questions for me, leave them in the comments. I`d be happy to answer!

Build an Amazon Scraper Using Your Chrome Profile

readysetscrape — Wed, 26 Nov 2025 11:16:15 +0000

How to build a simple Amazon scraper using your Chrome profile?

Look, I'll be honest with you - scraping Amazon isn't exactly a walk in the park. They've got some pretty sophisticated anti-bot mechanisms, and if you go at it the wrong way, you'll be staring at CAPTCHA screens faster than you can say "web scraping." But here's the thing: there's a clever way to do it that makes Amazon think you're just... well, you.

Let me walk you through how I built this scraper. Whether you're a business person trying to understand the technical side or a developer looking to build something similar, I'll break it down so it actually makes sense.

Come with me!

The Big Idea

This is where most people get it wrong. They fire up a fresh Selenium instance, maybe throw in some proxy rotation, and wonder why Amazon is blocking them after three requests. Sounds familiar? Here's the secret sauce: use your actual Chrome profile.

Think about it - your browser has your login sessions, your cookies, your browsing history. To Amazon, it looks like you browsing their site. Not some suspicious headless browser making requests at 3 AM.

At the very beginning, we need to find the folder where our Chrome profile is stored.
To do that, type chrome://version/ into the address bar.
There you'll immediately see the path to your profile.
For me, it looks like this:

C:\Users\myusername\AppData\Local\Google\Chrome\User Data\Profile 1

So the path we care about is:

C:\Users\myusername\AppData\Local\Google\Chrome\User Data\

For convenience, let's create a .bat file (my example is on Windows, but it works almost the same on Linux/macOS).

Inside the .bat file, add:

"C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9333 --user-data-dir="C:\Users\myusername\AppData\Local\Google\Chrome\User Data\"

Great! The most important part here is the port: 9333.
You can choose (almost) any number - I just picked this one.

Now, when you run the .bat file, Chrome will open with your profile already loaded.

Time to look at the code!
We want to connect Selenium to Chrome.
Let's grab Python by the head and get f*cking started!

class DriverManager:
    def connect(self):
        options = Options()
        options.add_experimental_option("debuggerAddress", f"localhost:{Config.CHROME_DEBUG_PORT}")
        self.driver = webdriver.Chrome(options=options)

See that debuggerAddress bit? That's connecting Selenium to your already running Chrome browser. You start Chrome with remote debugging enabled, and boom - Selenium can control your regular browsing session.

The beautiful part? If Amazon throws a CAPTCHA at you (and sometimes they will), you just solve it manually. The scraper waits patiently, and once you click those traffic lights or whatever, it continues on its merry way.

Simple but effective project stucture

I'm a big believer in keeping things clean and modular. Here's how I structured this:

src/
├── main.py              # app entry point
├── config.py            # all the boring configuration stuff
├── routes.py            # API endpoints
└── scraper/
    ├── driver_manager.py    # handles chrome connection
    ├── scraper.py           # scraping logic
    └── data_extractor.py    # parses and cleans the data

This singleton pattern ensures we're reusing the same browser connection. Why? Because starting up a new Chrome instance every time is expensive (both in time and resources), and more importantly, you lose all that precious session data.

The Scraper: where the MAGIC happens

Here's where we actually grab the data:

def search(self, query):
    response = self._get_response(f"https://www.amazon.com/s?k={query}&ref=cs_503_search")

    results = []
    for listitem_el in response.soup.select('div[role="listitem"]'):
        product_container_el = listitem_el.select_one(".s-product-image-container")
        if not product_container_el:
            continue

I'm using BeautifulSoup here because, let's face it, it's way more pleasant to work with than XPath or Selenium's built-in element finders. Once the page loads, I grab the HTML and let BeautifulSoup parse it. Simple as that.

Tip: Amazon's search results use a specific structure with div[role="listitem"]. This is pretty stable across their site variations. I learned this the hard way after my scraper broke twice because I was relying on class names that Amazon kept changing.

The Flask API - Make it happen!

I wrapped everything in a simple Flask API because, honestly, who wants to mess with Python imports every time they need to scrape something?

@api.route('/search', methods=['GET'])
def search():
    query = request.args.get('query', '')

    if not query:
        return jsonify({"error": "query required"}), 400

    try:
        driver = driver_manager.get_driver()
        scraper = Scraper(driver)
        result = scraper.search(query)
        return jsonify(result)
    except Exception as e:
        return jsonify({"error": str(e)}), 500

Now you can just:

curl "http://localhost:5000/search?query=mechanical+keyboard"

And get back nice, clean JSON with all the product details you need.

Yeah we did it!

Let me break down the advantages of using your own browser:

1. You're INVINCI... sorry! INVISIBLE (Mostly)
Using your real browser profile means you have:

Your actual cookies
Your login session (if you're logged in)
Your browsing history
Your browser fingerprint

All of this makes you look like a regular user, not a bot.

2. CAPTCHA? No way
When Amazon gets suspicious, you just solve the CAPTCHA like a normal person. The scraper waits, you click, life goes on.

3. Simple to maintain
No complicated proxy rotation, no headless browser detection workarounds, no constantly updating user agents. Just straightforward code that works.

4. Easy to debug
Because you can see the browser, debugging is trivial. Selector not working? Open the dev tools in your browser and figure it out.

Let's be real - Limitations

This approach is perfect for:

Personal projects
Building a prototype
Low-volume scraping
Understanding how Amazon's frontend works

But it's not great for:

High-volume production scraping
Running on servers (you need a desktop environment)
Parallel requests (one browser = one request at a time)
Completely automated, hands-off operation

For professional consider API

If you're running a business that needs reliable, high-volume Amazon data, you probably want something more robust. Managing your own scraping infrastructure gets complicated fast - you need proxies, CAPTCHA solving services, constant maintenance as Amazon changes their HTML...

For production use cases, I'd recommend checking out Amazon Instant Data API from our friends at DataOcean. They handle all the headaches of maintaining scrapers at scale, dealing with rate limits, rotating IPs, and keeping up with Amazon's changes. Sometimes paying for a good API beats maintaining your own infrastructure.

Any thoughts?

Building a scraper is part art, part science. The technical bits are straightforward once you understand them, but the real skill is in making architectural decisions that save you time down the road.

Using your own browser via remote debugging is one of those "why didn't I think of this sooner?" solutions. It's elegant, it works, and it keeps things simple.

Is it perfect? No. Will it scale to millions of requests? Also no. But for what it is - a clean, maintainable, easy-to-understand scraper that actually works - I'm pretty happy with it.

Now go forth and scrape responsibly. And seriously, if you need production-scale scraping, check out that DataOcean API or just contact me if your needs are much much than simple API could give you. Your future self will thank you.

Want the complete implementation?

👉 Get the full tutorial with all the code on my blog - it's free, no BS signup walls, just pure technical content.

The complete version includes:

Full price parsing implementation with all edge cases
Image URL manipulation tricks
Product details extraction code
Configuration best practices
Error handling patterns that actually work

Questions about web scraping?
Drop them in the comments or hit me up directly. I'm always happy to talk scraping strategies, Python architecture, or why BeautifulSoup is superior to XPath (fight me).

Happy scraping! 🚀