DEV Community

Cover image for Scrape Any Site in 20 Lines with Residential Proxies (Node.js & Python)
Aethyn Team
Aethyn Team

Posted on

Scrape Any Site in 20 Lines with Residential Proxies (Node.js & Python)

#node #python #tutorial #webscraping

Cross-posted from the Aethyn blog. I work on Aethyn — but the code here is standard-library stuff you can run against any proxy, so take the useful bits regardless.

Most proxy tutorials bury the code under 800 words of theory. Not this one. Terminal open, five minutes, working residential-proxy scraper by the end. We'll use the Aethyn SDK (proxy-builder-sdk), a thin wrapper over the proxy.aethyn.io gateway that assembles the proxy URL — country, city, sticky sessions, protocol — so you don't hand-build username strings.

1. Install 

npm install proxy-builder-sdk     # Node.js
pip install proxy-builder-sdk     # Python
Enter fullscreen mode Exit fullscreen mode

2. Authenticate

Use your aethyn-XXXXX username + password, or set AETHYN_USERNAME / AETHYN_PASSWORD and let the client read them.

from proxy_builder_sdk import AethynClient
client = AethynClient(username="aethyn-XXXXX", password="PASSWORD")
Enter fullscreen mode Exit fullscreen mode 
import { AethynClient } from "proxy-builder-sdk";
const client = new AethynClient({ username: "aethyn-XXXXX", password: "PASSWORD" });
Enter fullscreen mode Exit fullscreen mode

3. First proxied request

If the returned IP isn't yours, it works.

import requests
p = client.proxy(country="us")
r = requests.get("https://api.ipify.org?format=json", proxies=p.proxies)
print(r.json())
Enter fullscreen mode Exit fullscreen mode 
import { ProxyAgent, request } from "undici";   // npm i undici (axios/got also fine)
const p = client.proxy({ country: "us" });
const res = await request("https://api.ipify.org?format=json", {
  dispatcher: new ProxyAgent(p.url),
});
console.log(await res.body.json());
Enter fullscreen mode Exit fullscreen mode

client.proxy() rotates the IP on every request — ideal for stateless, high-volume reads.

4. Sticky sessions (same IP)

For logins, carts, and anything cookie-bound, pin one IP:

p = client.session(country="de", session="cart42", ttl=10)   # same IP, 10 min
Enter fullscreen mode Exit fullscreen mode 
const p = client.session({ country: "de", session: "cart42", ttl: 10 });
Enter fullscreen mode Exit fullscreen mode

5. City / ISP targeting (Elite) 

p = client.session(country="us", city="chicago", isp="comcast",
                   session="run1", ttl=30, tier="elite")
Enter fullscreen mode Exit fullscreen mode 
const p = client.session({ country: "us", city: "chicago", isp: "comcast",
                           session: "run1", ttl: 30, tier: "elite" });
Enter fullscreen mode Exit fullscreen mode

6. JavaScript sites: Playwright 

from playwright.sync_api import sync_playwright
p = client.session(country="fr", session="s1", ttl=15, tier="elite")
with sync_playwright() as pw:
    browser = pw.chromium.launch(proxy=p.for_playwright())
    page = browser.new_page(); page.goto("https://example.com")
    print(page.title()); browser.close()
Enter fullscreen mode Exit fullscreen mode 
import { chromium } from "playwright";
const p = client.session({ country: "fr", session: "s1", ttl: 15, tier: "elite" });
const browser = await chromium.launch({ proxy: p.forPlaywright() });
const page = await browser.newPage();
await page.goto("https://example.com");
console.log(await page.title());
await browser.close();
Enter fullscreen mode Exit fullscreen mode

7. SOCKS5 

p = client.proxy(country="jp", protocol="socks5")
Enter fullscreen mode Exit fullscreen mode 
const p = client.proxy({ country: "jp", protocol: "socks5" });
Enter fullscreen mode Exit fullscreen mode

Full mini-example: one page, three countries 

import requests
from proxy_builder_sdk import AethynClient

client = AethynClient()
url = "https://api.ipify.org?format=json"   # swap for your target
for cc in ["us", "de", "jp"]:
    p = client.proxy(country=cc)
    print(cc, "->", requests.get(url, proxies=p.proxies, timeout=30).json())
Enter fullscreen mode Exit fullscreen mode

Time-savers

Use proxy() (rotating) for stateless reads; switch to session() (sticky) the moment a cookie or login is involved. On a 403/429, rotate rather than retry-hammer — a rotating proxy does it on the next call, a sticky one when you change the session label. Verify your exit IP before trusting geo-specific data. And scrape public data only, respecting robots.txt, rate limits, and each site's terms — well-behaved scrapers get blocked far less.

New Aethyn accounts get a bandwidth allowance on both pools with no card and no time limit, so you can run every snippet above for free. Docs: aethyn.io/sdk.

Top comments (0)