DEV Community: proxymint

Web scraping architecture: components, patterns and proxies

proxymint — Mon, 02 Nov 2026 12:06:08 +0000

Originally published on proxymint.

Web scraping architecture splits a scraper into scheduler, fetcher, parser, storage and monitoring layers. How it changes with scale, and where proxies belong.

TL;DR

Five components, five failure modes. A scraping system splits into a scheduler and queue, a fetcher with its proxy layer, a parser with validation, storage, and monitoring. Each one gets its own metrics and retry rule.
Rate limits live in the scheduler. A concurrency budget per target domain, enforced centrally, keeps every worker inside it. Adding exit IPs never raises that budget.
The proxy layer is an interface, not a loop. Workers ask for an egress by job type and country; rotation is chosen per job shape and DNS resolves at the proxy with CONNECT or socks5h.
Some errors mean stop. A 429 means wait as long as Retry-After says; a 403 or challenge page means the job for that target stops and a person reviews access.

Key takeaways

Check for an official API or data export first. Scraping is the fallback for public pages where no such route exists.
Store raw responses before parsing. A broken selector then costs a replay over stored pages, with no new requests to the target.
Enforce robots.txt and per-domain budgets before the queue. Cache robots.txt for at most 24 hours, as RFC 9309 says, and share one limiter across all workers.
Give each job shape its own egress profile. A new IP per request for independent pages, a timed session for paginated flows, a static address for long logins on accounts the team owns.
Alert on change per layer. Parse yield, 429 share per domain and data freshness catch problems the day they start.

Read the full guide on proxymint, with the commands, tables and sources.

Puppeteer proxy setup: launch flag, auth, contexts, SOCKS5 and rotation

proxymint — Mon, 02 Nov 2026 08:06:05 +0000

Originally published on proxymint.

Set a Puppeteer proxy with --proxy-server or per browser context, authenticate with page.authenticate, fix 407 and SOCKS5 errors, and rotate by context.

TL;DR

Puppeteer takes a proxy per browser or per context. The --proxy-server launch flag covers the whole browser, and createBrowserContext({ proxyServer }) gives one isolated context its own exit inside the same browser.
Credentials go through page.authenticate, never the flag. Chrome does not use credentials embedded in proxy settings, and a user:pass@ URL in --proxy-server fails with net::ERR_NO_SUPPORTED_PROXIES.
A missing authenticate call looks like a network error. Current Chrome reports net::ERR_INVALID_AUTH_CREDENTIALS on HTTP pages and net::ERR_TUNNEL_CONNECTION_FAILED on HTTPS pages, not a 407 status.
SOCKS5 with a password does not work in Chrome. Chrome supports no SOCKSv5 authentication methods, so a password-protected SOCKS5 server fails with net::ERR_SOCKS_CONNECTION_FAILED whatever page.authenticate says.
Chrome allows 32 simultaneous connections through an HTTP/1.1 proxy. Heavy parallel work in one browser queues inside Chrome before the proxy sees it.

Key takeaways

Call page.authenticate on every page before goto. Cached credentials can make a forgotten call pass in testing and fail on a fresh context in production.
Give each identity its own context. Rotate by closing the context and opening the next on a fresh sticky session, so the IP, cookies and cache change together.
Use the HTTP endpoint when SOCKS5 needs a password. HTTPS pages still run end to end through the CONNECT tunnel.
Check the exit before the job runs. Assert the country inside the page, and set Accept-Language for the market whose pages the job reads.
Block images, fonts and media before sizing the plan. Request interception works alongside page.authenticate and cuts the bytes a per-GB tier bills.

Read the full guide on proxymint, with the commands, tables and sources.

Browser profile proxy setup: one proxy, WebRTC and DNS

proxymint — Sun, 01 Nov 2026 12:06:06 +0000

Originally published on proxymint.

Set up one proxy per browser profile: own credentials per profile, the proxy type for each job, and how to check WebRTC, DNS and language before login.

TL;DR

A browser profile proxy setup gives each profile its own proxy. The profile keeps its own cookies, saved logins and settings such as language; the proxy decides where its traffic leaves from: IP, network, country and city.
One profile, one proxy, one set of credentials. Each client's sessions stay on one consistent address, and no profile carries another client's proxy login.
Pick the tier by the job. ISP for long-lived logins, residential with a sticky timer for logged-out ad checks, mobile for app-first accounts, datacenter IPv4 for QA on your own sites.
WebRTC and DNS are the usual leaks. RFC 8828 notes that HTTP and most SOCKS proxies carry no UDP, so set WebRTC to proxy-only, and confirm that names resolve on the proxy side.
Verify inside the launched profile before the first login. Check exit IP, language, WebRTC, DNS resolver and IPv6, and run it again after any exit change.

Key takeaways

Assign exactly one proxy to each profile. Name the profile after the client or market, paste host, port, username and password, and use the http:// scheme.
Keep credentials per profile. A static ISP IP comes with its own username and password; on per-GB tiers give each profile its own port and each client its own order.
Set the language for the market. The profile language drives navigator.language and Accept-Language; set it to the market whose pages the job needs.
Switch WebRTC to proxy-only or address replacement. Never leave the default that routes WebRTC from the host network.
Use the platform's own access route first. Meta partner access and Google Ads manager accounts often remove the need for a separate profile holding a client's login.

Read the full guide on proxymint, with the commands, tables and sources.

Backconnect proxy: one gateway, many exit IPs, how to test

proxymint — Sun, 01 Nov 2026 08:06:04 +0000

Originally published on proxymint.

A backconnect proxy is one gateway address that routes your traffic to exit IPs from a pool. Gateway vs exit, rotation, proxy lists, and how to test one.

TL;DR

A backconnect proxy is one gateway address with a pool of exit IPs behind it. Your client configures the gateway, and the target only ever sees the exit.
The port is the unit you work with. On a list-based gateway every line shares one host and one login and differs by port, so parallel sessions mean more ports, not more logins.
Rotation is a setting, not a different product. The same gateway rotates per request, on a 5-60 minute timer, or holds an exit while the device stays online.
Test before you scale. A short script shows whether rotation follows requests or connections, how ports spread over exits, and how long sticky ports hold.

Key takeaways

Pick the mode by the longest step that must share an IP. Then write the client to survive an early IP change, because a device can drop before the timer ends.
Give each browser context or worker its own port. Keep per-request rotation away from live pages that load many assets.
Use http:// or socks5h:// in the proxy URL. Names then resolve on the proxy side, and HTTPS targets stay encrypted inside the tunnel.
Move identity that must hold for days to an ISP address. A gateway gives short windows, not a fixed IP for weeks.

Read the full guide on proxymint, with the commands, tables and sources.

What network error sent you looking in the wrong place?

proxymint — Sat, 31 Oct 2026 20:02:59 +0000

The network bugs that eat a whole day tend to have the same shape: the error points at the wrong layer.

A 407 that looks like the site blocking you, when it's the proxy refusing a password with an unencoded @ in it. A scraper that works on a laptop and times out on a server because the server's DNS resolver is somewhere else entirely. A curl call that exits 0 while the request never reached the site. A redirect loop that only Chrome sees, because of an HTTPS record nobody knew was published.

So we're curious what yours was. What's the network or proxy error that sent you looking in completely the wrong place, and what was it actually?

Bonus points for the one-line fix at the end. We'll read every answer and reply to each one.

curl through a proxy can exit 0 when the login fails

proxymint — Sat, 31 Oct 2026 18:41:59 +0000

Small thing that probably bites a lot of cron jobs out there: curl through a proxy, wrong password, and $? is 0.

No crash, no stderr. The output file just has an empty 407 in it where the page should be, and whatever reads that file next is going to have a bad day.

I wanted to know exactly when this happens, so I wrote a fake proxy that refuses everyone and pointed curl 8.7.1 at it.

# p407.py - a fake proxy that refuses every login
import http.server, socketserver

class H(http.server.BaseHTTPRequestHandler):
    def refuse(self):
        self.send_response(407)
        self.send_header("Proxy-Authenticate", 'Basic realm="p"')
        self.send_header("Content-Length", "0")
        self.end_headers()
    do_GET = do_CONNECT = do_HEAD = refuse

socketserver.TCPServer(("127.0.0.1", 18407), H).serve_forever()

http:// target: exit 0

curl -s -o out.html -w '%{http_code}\n' -x http://user:pass@127.0.0.1:18407 http://example.com/
echo "exit $?"

407
exit 0

Yep. With a plain http URL curl hands the whole request to the proxy, the proxy says 407, and that comes back like any other response. curl got an answer, job done. Bad password, an @ in the password nobody encoded, plan ran out, doesn't matter, your script sees a clean exit.

-f fixes it

curl -sf -o out.html -x http://user:pass@127.0.0.1:18407 http://example.com/
echo "exit $?"

exit 22

-f tells curl that 400 and up is a failure, so you get 22. --fail-with-body does the same but keeps the body, handy when you want to see what the proxy actually said.

https:// target: fails anyway, different number

curl -sS -o /dev/null -x http://user:pass@127.0.0.1:18407 https://example.com/
echo "exit $?"

curl: (56) CONNECT tunnel failed, response 407
exit 56

Here curl has to ask the proxy for a tunnel first (CONNECT), and the 407 shows up before TLS even starts, so there's nothing to pretend about. 56 on 8.7.1. We got 7 for the same thing on 8.21.0, so if your script matches exact numbers, don't. Non-zero means it broke.

Honestly the https case isn't the scary one. The scary one is the cheap health check or the old scraper hitting an http endpoint, because that's where it fails quietly for days.

The fix I'd use

Check the status code too, not only the exit code:

code=$(curl -s -o out.html -w '%{http_code}' -x "$PROXY" "$URL")
rc=$?
if [ "$rc" -ne 0 ] || [ "$code" -ge 400 ]; then
  echo "failed: exit=$rc status=$code" >&2
  exit 1
fi

Or just -f plus set -e if you like it short.

Two neighbours of this bug:

SOCKS5 can't give you a 407 at all, there's no HTTP in between. A bad login there is curl: (97) User was rejected by the SOCKS5 server. Other number, same problem.
p@ss in a proxy URL has to be p%40ss. Otherwise curl cuts the URL at the wrong @, sends a username the proxy has never heard of, and you're staring at a 407 for a password you know is right.

We wrote up the rest of curl-through-a-proxy (socks5 vs socks5h and DNS leaks, --noproxy, timing each hop with -w, a five-command check that tells you which layer broke) here: curl proxy: -x, auth, SOCKS5, timing and exit codes.

Disclosure: I work on proxymint, a proxy service. None of the above depends on whose proxy it is.

How many proxies do I need? A sizing formula by job

proxymint — Sat, 31 Oct 2026 12:06:05 +0000

Originally published on proxymint.

How many proxies you need depends on the job: requests per hour, concurrent sessions, accounts or GB. Formulas, a pilot test and a worked example for each job.

TL;DR

The count depends on the billing model. Static ISP and datacenter IPv4 sell IPs, so the number is literal; rotating residential and mobile sell GB, so the pool supplies the IPs and the job sizes traffic and sessions.
For collection, size by concurrency, not by a per-IP rate. Rented IPs equal requests in flight divided by the connections one IP carries, at least one per country, plus a 10-30% reserve.
The target sets the pace for the whole job. Its terms, API limits and robots.txt decide it; a 429, block page or challenge in the pilot means slow down or stop, not add IPs.
Accounts need one static IP per login location, not per account. Teams using the platform's own team or partner access from their own office often need none.

Key takeaways

Name the billing model first. Decide whether the order buys IPs or GB before running any formula, because the question changes with it.
Size for peak and for concurrency. Requests in flight equal requests per second times seconds per request, and sticky sessions need one exit per session running at once.
Run a pilot before ordering. Send the job's own pace from one IP on public pages the target permits, log status codes and response times, and stop at the first 429 or challenge.
Treat a block as a stop signal. More IPs do not change a target's answer; check its terms and official API before going further.
Measure bytes per page on per-GB tiers. A 20-page sample through the real client beats any median, and the fetch mode moves the bill far more than the pool does.

Read the full guide on proxymint, with the commands, tables and sources.

Cheap datacenter proxies: billing, sizing and where they fit

proxymint — Sat, 31 Oct 2026 08:06:06 +0000

Originally published on proxymint.

Cheap datacenter proxies are billed per IP or per GB, but value is set by how many requests return data. How to measure it, size an order and pick the tier.

TL;DR

Count useful responses, not addresses. A port is billed per month, so what it returns is requests times success rate; at a 25% success rate it takes four requests to get one useful response.
On lenient targets a port's bill does not follow volume. A dedicated IPv4 port is billed per month with no per-GB meter, so a million requests are billed the same as ten thousand.
Size by concurrency and country, not by the target's limit. A target's rate limit covers the whole job, so more ports never buy more pace on one site.
Datacenter addresses are labelled as hosting. IP databases flag hosting ranges, so platforms with anti-bot scoring and strict retail challenge them on sight.

Key takeaways

Run a small-port test first. Measure 2xx, 403, 429 and challenge rates on 50-100 real pages before buying bulk.
Size by concurrency, not by the target's limit. Ports needed equals parallel workers divided by workers per port, per country; the job's total pace stays inside what the target allows.
Use ports for heavy pages. A port has no per-GB meter, so full renders at the 2.6 MB median page weight stay flat-priced.
Match the tier before buying. Targets that score hosting ranges fit rotating residential per GB, and one-address work fits static ISP.

Read the full guide on proxymint, with the commands, tables and sources.

Proxy error codes: which hop sent it and how to fix it

proxymint — Fri, 30 Oct 2026 12:06:05 +0000

Originally published on proxymint.

Proxy error codes decoded by hop: which ones the proxy sends, which come from the target, how to read them with curl -v, and which ones are safe to retry.

TL;DR

The hop matters more than the number. A request through a proxy crosses the client, the gateway, the exit and the target, and the same 502 means different things depending on which one sent it.
curl -v shows the phase. On an https:// target an error that answers the CONNECT came from the proxy side; anything after 200 Connection established came from the target.
Some failures have no status code. Refused connections, timeouts and SOCKS5 replies surface as client errors such as curl exit codes 5, 7, 28, 56 and 97.
Retry only what can pass. 429, 503 and gateway-phase 502 or 504 are worth a backoff or a new exit; 400, 407 and refused connections are setup errors.

Key takeaways

Run the failing request with curl -v first. It tells you whether the error came back on the CONNECT or inside the tunnel, which decides who can fix it.
Read Proxy-Status and Retry-After when present. RFC 9209 lets a gateway name the failing hop and cause; Retry-After tells you how long to wait on 429 and 503.
Fail fast on 400, 407 and refused connections. Log the exact proxy string and error, fix the setting, and keep these out of the retry loop.
Treat repeated 403s as a fit question. Slow down, check the target terms and API, and match the proxy category to the network type the target accepts.

Read the full guide on proxymint, with the commands, tables and sources.

How to choose a residential proxy provider: a buyer's checklist

proxymint — Fri, 30 Oct 2026 08:06:06 +0000

Originally published on proxymint.

How to choose a residential proxy provider: sourcing and consent, pool depth where you need it, sessions, geo accuracy, real cost per GB and a test plan.

TL;DR

Five checks decide a residential proxy provider. Sourcing and consent, live supply in your cities, session controls, the real cost at your band, and a small paid test on your own target.
Consent needs a statement, not a badge. Ask how devices joined the pool, whether owners agreed, and whether they can leave.
City targeting is only as good as the target's IP database. MaxMind states about 66% accuracy for US cities within 50 km, against about 99.8% at country level.

Key takeaways

Write the job down first. Targets, cities, session length and monthly GB decide which provider and which tier fit.
Get the sourcing answer in writing. Score "ethically sourced" with no consent statement as a no.
Price at the GB you will actually buy. Include GB expiry and refund terms, not only the banner rate.
Test before a large order. Measure success rate, concurrency, geo and exit reuse on a target you are permitted to access.
Check that residential is the right tier. Datacenter suits lenient high-volume targets, static ISP holds one identity for days, and mobile suits mobile-only platforms.

Read the full guide on proxymint, with the commands, tables and sources.

Residential proxy pricing: what a GB buys for your job

proxymint — Thu, 29 Oct 2026 12:06:04 +0000

Originally published on proxymint.

Residential proxy pricing per GB explained: what a GB buys in pages, traffic per 1,000 pages, expiry and run-outs, and when another tier fits the job better.

TL;DR

proxymint sells residential per GB. Packages are one-off with no renewal, and the rate drops as the package grows.
A GB means pages only once page weight is known. At the 22 KB median HTML size one GB is about 45,000 pages; at the 1.8 MB median mobile inner page it is about 550 full renders.
1,000 pages move as many GB as one page weighs in MB. Full renders at 1.8 MB move about 1.8 GB per 1,000 pages, and the fetch method moves that far more than the package size does.
Nothing renews and no expiry is set on the GB. A package bought for a quarterly job is still on the balance for the next run.

Key takeaways

Budget in pages, not GB. Fetch 20 representative pages through the real client, average the bytes and multiply by the page count, plus 20-30% headroom.
Pick the fetch method before the package size. HTML or JSON instead of a full render cuts the traffic per page by one or two orders of magnitude.
Size the package to the job. Read the rate for that quantity on the pricing page.
Buy one package per job. One package sized to the job means one order and one balance to watch.
Match the tier to the job. Lenient bulk work fits datacenter ports, and fixed-address work fits static ISP IPs.

Read the full guide on proxymint, with the commands, tables and sources.

Mobile proxy pricing: what one carrier GB buys

proxymint — Thu, 29 Oct 2026 08:06:06 +0000

Originally published on proxymint.

Mobile proxy pricing explained: what a carrier GB buys, how to size a job before you pay, cost per successful request, and which jobs call for mobile IPs.

TL;DR

Mobile proxies are usually sold per GB. On proxymint they come in one-off packages, the rate falls as the package grows, and the full table is on the pricing page.
The rate reflects carrier supply and trust. Carriers share one public address across many subscribers with CGNAT, so platforms that score network origin treat carrier ranges carefully.
Bytes drive the bill. A full mobile page render moves roughly 80 times the bytes of its HTML, and every retry is billed again.
Measure cost per success. Success rate and bytes per attempt on the real target decide what a job actually costs.
Match the tier to the target. Mobile fits app-first platforms, mobile ad checks and QA on carrier routing; catalogs and price checks usually clear on residential.

Key takeaways

Size the job first. GB needed = requests x bytes per request x a retry factor, measured on a small sample.
Measure success on the target. A platform that scores mobile origin shows it in the success rate of a small test.
Test both tiers on a small package. Run the same 200 to 500 permitted requests through each and record success and bytes.
Route desk-based account work to static ISP. Fixed ISP addresses are billed per IP per month; mobile fits accounts that live in an app.
Buy one package per job. A package sized from the test covers the whole run, with no renewal and no expiry set on the GB.

Read the full guide on proxymint, with the commands, tables and sources.