The proxy industry needs you to never open the network tab

I run 75 scrapers in production. Three of them do any fingerprint spoofing. Maybe five use residential proxies. The rest run on plain datacenter IPs or no proxy at all, and they have been running for months.

If you learned scraping from blog posts, that number probably sounds wrong to you. Every tutorial you have read starts the same way: sign up for a residential pool, install a stealth browser, randomize your fingerprint, throttle like a human. Then, on step five, you finally get to look at the actual website.

That order is backwards, and it is backwards on purpose. Proxy companies write most of the scraping content on the internet. They were never going to write "you probably do not need us."

The scraper with the $80 a month costume

Last month my guy sent me his Greenhouse job board scraper to fix. It had everything. Puppeteer with the stealth plugin. Rotating residential proxies. Randomized mouse movements between actions. Human-like typing delays.

It still kept dying.

So I did the thing nobody had done: opened the page in a normal browser with devtools up. The entire job list was sitting in one XHR request to a public JSON endpoint. No auth. No cookies. A rate limit so loose I never managed to hit it.

I deleted basically his entire codebase and replaced it with a fetch call. It has not broken since. He had been paying for proxies for months to hit an endpoint that does not care who you are.

This was not a rare lucky case. This is most cases.

The 20 minute method

What I do on every new target, before writing a single line of code:

1. Open the network tab, filter to XHR/fetch.
2. Reload the page. Click around. Paginate. Search.
3. Find the request that returns the actual data. It is usually JSON and usually obvious.
4. Right click, copy as cURL.
5. Paste it in a terminal and start deleting headers one at a time. Rerun after each delete.

Whatever survives step five is your scraper. Most of the time the answer is a user agent header and nothing else. Sometimes a referer. Occasionally one cookie you can grab with a single cheap request first.

Twenty minutes. No browser automation, no proxy bill, and the result is faster and more stable than any puppeteer setup, because JSON endpoints change way less often than HTML markup does.

Half the "protected" sites out there are react apps sitting on a wide open API. The frontend is the security theater. The data is just there.

When you actually need the heavy stuff

I am not saying anti detect tooling is fake. I am saying it is a last resort that got marketed into a first step. You genuinely need it when:

• The protection sits on the API itself, not the page. Turnstile or a challenge token attached to the data endpoint.
• The site checks TLS fingerprints, so plain curl or node fetch gets flagged before your headers even matter.
• Sessions are bound to browser fingerprints, ticketing platforms mostly.
• You need real logged-in browsing at scale.

In my portfolio that is maybe 1 target in 10. For those, sure, pay for the pool. For the other 9, the network tab was free the whole time.

The tell

You can tell how someone learned scraping by what they reach for first. People who learned from vendor blogs can configure a fingerprint rotator but have never stripped a cURL command down to its minimum headers. People who learned by breaking things do it the other way around.

Try the boring thing first. It costs nothing, and most of the time it is the whole job.

I write about this stuff and post scraping methods on r/scrapingtools if you want more of it.