How to Migrate Your Scraper Off NetNut in Under an Hour

#webscraping #python #proxy #api

In July 2026, the FBI and Google disrupted NetNut's residential proxy network. According to reporting from Krebs on Security and SecurityWeek, investigators found that a large share of NetNut's roughly two-million-device IP pool had been built on compromised consumer devices rather than fully informed, consented participation.

If you had PROXY = "http://user:pass@resi.netnut.io:5959" sitting in a config file somewhere, here's the fast path to swapping it out, plus a couple of scripts to make sure the replacement actually works before you trust it in production.

Step 1: test the new provider against your real targets first

Don't swap and hope. Pull together your actual target URLs — the ones your scraper hits day to day, not a generic test site — and run a quick success-rate check before you migrate anything for real.

import requests

TARGETS = [
    "https://example.com/product/1",
    "https://example.com/product/2",
    # add 20-50 of your real target URLs here
]

def test_provider(proxy, targets, timeout=10):
    ok = 0
    for url in targets:
        try:
            r = requests.get(
                url, proxies={"http": proxy, "https": proxy}, timeout=timeout
            )
            if r.status_code == 200:
                ok += 1
        except requests.RequestException:
            pass
    return ok, len(targets)


if __name__ == "__main__":
    proxy = "http://user:pass@residential.newprovider.com:8000"
    ok, total = test_provider(proxy, TARGETS)
    print(f"{ok}/{total} succeeded ({ok/total:.0%})")

A proxy that looks great on a generic connectivity test can still fail badly on the specific sites you actually scrape, SERPs and retail sites in particular tend to fingerprint harder than a plain "does it connect" check will catch. Test on your own targets, not a benchmark someone else picked.

Step 2: swap it in Scrapy

If you're on Scrapy, the proxy usually lives in one of two places: a single value in settings, or a middleware that sets it per-request.

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 1,
    "myproject.middlewares.ProxyMiddleware": 100,
}

# Before (NetNut)
# PROXY = "http://user:pass@resi.netnut.io:5959"

# After — same shape, new endpoint
PROXY = "http://user:pass@residential.newprovider.com:8000"
python
# middlewares.py
from myproject.settings import PROXY

class ProxyMiddleware:
    def process_request(self, request, spider):
        request.meta["proxy"] = PROXY

That's the whole migration for most Scrapy setups, one string changes, nothing else in the spider logic needs to know or care.

Step 3: swap it in Playwright

Browser automation proxies get set at launch time instead of per-request.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(
        proxy={
            "server": "http://residential.newprovider.com:8000",
            "username": "user",
            "password": "pass",
        }
    )
    page = browser.new_page()
    page.goto("https://example.com")

Same idea for Puppeteer if you're on the JS side, the proxy goes into launch()'s args, not scattered through your page logic.

What to actually check before picking who's next

The technical swap is the easy part. The part worth spending real time on is picking a provider whose sourcing you can actually verify, since "residential proxy" is a claim, not something a trial run confirms. Quickly, before you commit:

Ask for the sourcing mechanism specifically - a named SDK partnership and a public acceptable-use policy, not a reassurance.
Match the pricing model to your actual traffic pattern - per-GB is fine for steady usage, flat per-IP is more predictable if your volume swings.
Weigh operating history like you would for any other infrastructure dependency.

If you're specifically looking for a NetNut alternative rather than just picking whatever's cheapest, Squid Proxies is one worth running the scripts above against: its residential network has been operating for sixteen-plus years, sources IPs through direct, consent-based SDK partnerships, and prices residential proxies at $0.75–$1.50/GB. That's still just a claim until you've tested it on your own targets, which is exactly what step 1 is for, regardless of who you end up picking.

Recap

Test the candidate provider against your real target URLs, not a generic benchmark.
Swap the proxy string in Scrapy settings or the Playwright/Puppeteer launch config, usually a one-line change.
Actually check sourcing, pricing model fit, and operating history before you commit, not after.

Total time, most setups: closer to twenty minutes than an hour. The title's got some slack built in for whoever's target sites are pickier than average.

DEV Community