DEV Community: Andrew

Proxy DNS leak test: check and fix it in your own client

Andrew — Tue, 13 Oct 2026 08:05:56 +0000

Originally published on proxymint.

A proxy DNS leak test you run through your own client: where curl, requests, httpx, Playwright and Node resolve names, what a leak exposes, and each fix.

TL;DR

A proxy carries the connection, and the DNS lookup can still happen locally. When the client resolves the target hostname itself and hands the proxy an IP, that lookup leaves through the local network.
HTTP proxies resolve at the proxy by construction. The CONNECT request carries the hostname, so curl, requests, Chromium and Go make no local lookup for the target.
SOCKS5 depends on the client. curl, Python requests and Node socks-proxy-agent resolve locally for socks5:// and at the proxy for socks5h://, while httpx, Chromium and Go send the hostname for both.
A leak rarely throws an error. It leaks the target list to the local resolver, returns CDN answers for the wrong city and shows a resolver that does not match the exit.
There are two halves to test. A local capture shows whether the client sends names; a far-side test shows where the provider resolves them, which only the provider can change.

Key takeaways

Use an http:// proxy URL for HTTP and HTTPS jobs. It sends the hostname to the proxy in every standard client and removes the SOCKS DNS question.
Write socks5h:// wherever a job must run over SOCKS5. In curl, requests and socks-proxy-agent the h is the difference between local and remote resolution.
Run a port 53 capture once per client and after every library upgrade. Only the proxy's own hostname should appear, and a DoH client needs the Python hook or a NetLog instead.
Test the far side before a geo-sensitive job scales. Compare the resolver's country with the exit's, for every provider, and ask the provider when they never match.

Read the full guide on proxymint, with the commands, tables and sources.

US mobile proxies: carriers, targeting and exit checks

Andrew — Mon, 12 Oct 2026 12:05:54 +0000

Originally published on proxymint.

US mobile proxies exit through US carrier networks. How CGNAT and the three national carriers shape them, how precise targeting is, and how to check the exit.

TL;DR

A US mobile proxy exits from a US carrier network. The address belongs to a mobile carrier's ASN and IP databases label it as a cellular connection in the United States, the address class a phone on 4G or 5G uses.
Three nationwide carriers carry the market. The FCC names AT&T, T-Mobile and Verizon as the nationwide facilities-based providers, and prepaid brands and MVNOs run on their networks.
Carrier addresses are shared. More than 90% of cellular networks measured ran carrier-grade NAT (Richter et al., IMC 2016), so one public IP fronts many subscribers, and Cloudflare found CGNAT addresses rate limited three times more often (October 2025).
Region is the reliable level on cellular. MaxMind says mobile geolocation typically resolves to a broad region rather than a city or postal code, so city and ZIP work also sets location inside the app or site.

Key takeaways

Check the connection type, not the label. Echo the exit, confirm a carrier ASN and a mobile or cellular flag in two databases, and confirm DNS resolves on the proxy side.
Target the region for state work. Use country for national jobs and region for state campaigns, and pair a city exit with the app's own location setting.
Match rotation to the flow. Per-request rotation for stateless ad and page checks, a 5 to 60 minute timer for short flows, sticky for a signed-in session that survives an address change.
Make the client look like a phone in the same place. Set en-US, the IANA time zone of the exit region and a mobile device profile in a fresh browser context per market.
Send other jobs to the tier built for them. Rotating residential covers US price monitoring, static ISP holds one US identity for weeks, and datacenter carries volume on lenient targets.

Read the full guide on proxymint, with the commands, tables and sources.

How to scrape Google Maps: Places API first, then scraping

Andrew — Mon, 12 Oct 2026 08:05:53 +0000

Originally published on proxymint.

How to scrape Google Maps: how the Places API is billed now, what Google's terms allow, and a scraping method for public business data with city-level proxies.

TL;DR

Try the Places API before scraping. Since March 1, 2025 the Google Maps API free tier is a monthly cap per SKU, so a job of a few hundred places with phone and website often fits inside it.
Google's terms say no to scraping on both routes. Section 3.2.3(a) of the Maps Platform terms names bulk downloads of places information, and the end user terms ban mass downloads and bulk feeds of Maps content.
The API does not give you a database either. Only place IDs may be stored indefinitely and coordinates for up to 30 days, so the API cannot back a permanent lead list.
Proxies only matter on the browser route. Residential exits with city targeting fit it, because the exit can sit in the map area; datacenter IPv4 is located by country only, and mobile is built for carrier-origin targets rather than a catalog-style job.

Key takeaways

Write the field mask first. List the fields the job needs before choosing a route: phone, website and rating put a Text Search request on the Enterprise SKU, billed per 1,000 requests after 1,000 free a month.
Measure places per request. Log how many new places each request returns on the first 100 calls; that ratio sets how many billable requests the job needs before deciding between the API and a browser.
Leave reviews and personal contact data alone. Reviewer names and a sole trader's mobile number are personal data, and Google's terms name user reviews explicitly.
Treat a challenge page as a stop. Slow down, narrow the job or move it to the API; never push through with a captcha solver or a fresh exit.

Read the full guide on proxymint, with the commands, tables and sources.

How to scrape Amazon: Python, the law and official APIs

Andrew — Sun, 11 Oct 2026 08:05:48 +0000

Originally published on proxymint.

Scrape public Amazon product and price data with Python: what the terms and courts say, the official API routes, polite rate limits and where proxies fit.

TL;DR

There are three routes to Amazon data. The official APIs if you qualify (Creators API for Associates, SP-API for sellers), your own scraper on public logged-out pages, or a bought dataset.
Amazon's terms prohibit scraping. The Conditions of Use exclude any collection of product listings or prices and any use of robots or data mining tools, and robots.txt bars 99 named agents, Scrapy included, from the whole site.
Courts separate access from contract. US rulings since 2021 treat logged-out collection of public pages as mostly outside computer-crime law, but contract, database right and the GDPR still apply.
The code should stop, not evade. A 503, a robot-check page or a sign-in redirect means log it, back off and end the run after a short streak.
Most small jobs need no proxy. A few hundred ASINs a day at a slow pace can run from one IP; residential exits fit other countries' marketplaces, and volume past one polite exit belongs on the official APIs or a bought dataset.

Key takeaways

Check the official APIs before building anything. Associates with 10 qualifying sales in 30 days can use the Creators API, and sellers get pricing data from SP-API.
Stay on public, logged-out pages. Product pages and search pages only: no cart, no sign-in, no reviews behind the login wall and no reviewer names.
Set your own slow pace and log every fetch. One request every few seconds per exit with jitter, one at a time, with timestamp, marketplace and status on every row.
Match marketplace, language and exit country. Prices and currency follow the domain and the request's location, so all three have to agree.
Measure MB per page before buying traffic. GB needed is pages times MB per page divided by 1,024, and HTML-only fetches are the biggest cost lever.

Read the full guide on proxymint, with the commands, tables and sources.

Germany proxies: targeting, sessions and exit checks

Andrew — Sat, 10 Oct 2026 12:05:45 +0000

Originally published on proxymint.

Germany proxies give a German exit IP. Which proxy type fits, how state and city targeting works, why DE is the code, and how to check the exit before a job.

TL;DR

A Germany proxy exits from an address placed in Germany. It can be a home line, a static ISP address, a mobile carrier or a datacenter server, and the source decides how a target classifies it.
Lookups answer DE, not GER. DE is the ISO 3166-1 code for Germany; GER is a sports code and GE is Georgia.
State matters for holidays, delivery and local search. German consumer prices already include VAT, so a plain price check rarely needs more than a country exit.
The IP is one German signal of several. Europe/Berlin with summer time, a de-DE locale and number format, and DNS resolved on the proxy side have to match the exit.

Key takeaways

Pick the type by requirement. Rotating residential for broad consumer coverage, static ISP for one identity over weeks, mobile for app-first platforms, datacenter for lenient targets.
Pick the level from the page. Country for national pages, state when holidays or regional offers change it, city only for local search, store stock or city ads.
Match on DE. Normalise Germany, Deutschland and GER to DE before comparing an exit with the order.
Plan for the daily reconnect. Some German home lines, o2 DSL and fibre among them, are cut once every 24 hours and come back on a new address, so long residential sessions should survive a new IP.
Parse prices with the page locale. German pages write 1.299,99 €, and they show a unit price next to the total.

Read the full guide on proxymint, with the commands, tables and sources.

407 Proxy Authentication Required: causes and client fixes

Andrew — Sat, 10 Oct 2026 08:05:47 +0000

Originally published on proxymint.

A 407 means the proxy, not the site, rejected or never got your login. Causes in the order to check, and the fix for curl, Python, Node, browsers and apt.

TL;DR

A 407 comes from the proxy, not the website. The proxy got no credentials or the wrong ones, answered with a Proxy-Authenticate challenge, and never forwarded the request to the site.
The challenge scheme tells you which problem you have. Basic means a provider or simple proxy that wants a username and password. NTLM or Negotiate means a corporate proxy that wants a Windows or domain login.
On HTTPS the 407 answers the CONNECT, so there is no response object. Clients raise errors such as "Tunnel connection failed: 407" or "CONNECT tunnel failed, response 407", and credentials must ride on the CONNECT itself.
Most 407s are placement or encoding. Credentials in a field the client ignores, a requests proxies dict without an https key, or an unencoded @, : or % in the password cause most cases.

Key takeaways

Run curl -v through the proxy with and without -U first. If -U works, fix how the client sends credentials; if it fails, fix the credentials or the account.
Percent-encode the username and password in code. Use urllib.parse.quote with safe set to empty, or encodeURIComponent, because a % followed by two hex digits silently changes the password.
Use each client's own credential fields. Playwright needs username and password fields, Puppeteer needs page.authenticate, and Java needs jdk.http.auth.tunneling.disabledSchemes cleared.
Never retry a 407 in a loop. Treat it as a configuration error, alert, and keep retries for 429 and 5xx responses.

Read the full guide on proxymint, with the commands, tables and sources.

UK residential proxies: regions, cities and sessions

Andrew — Fri, 09 Oct 2026 12:06:01 +0000

Originally published on proxymint.

UK residential proxies route through British home broadband. How UK targeting works, why GB is the country code, which session fits and how to check the exit.

TL;DR

A UK residential proxy exits from a British home connection. The address belongs to a consumer ISP network such as BT, Sky, Virgin Media or TalkTalk, so databases classify it as residential and place it in the UK.
Lookups answer GB, not UK. GB is the ISO 3166-1 code for the United Kingdom, so configs, tests and filters should match on GB.
Region and city matter for delivery, devolved rules and local search. UK consumer prices already include VAT, so a plain price check rarely needs more than a country exit.
The IP is one UK signal of several. Europe/London with British Summer Time, an en-GB locale and DNS resolved on the proxy side have to match the exit.

Key takeaways

Pick the level from the page. Use country for national pages, region when delivery or devolved rules change, and city only for local search, store stock or city ads.
Match on GB. Normalise United Kingdom, UK and England labels to GB before comparing an exit with the order.
Check exits in two databases first. Echo the IP, confirm a consumer ISP network and the city in two lookups, and confirm DNS resolves on the proxy side.
Set Europe/London and en-GB per context. Never hardcode a +00:00 offset; it is wrong during British Summer Time.
Use static ISP for a long-lived UK identity. A home device can go offline; one assigned UK ISP address holds for the whole term.

Read the full guide on proxymint, with the commands, tables and sources.

Why proxies trigger captchas, and what reduces them

Andrew — Fri, 09 Oct 2026 08:05:46 +0000

Originally published on proxymint.

Why proxies trigger captchas: the risk score behind each challenge, the inputs your proxy tier changes, and what a legitimate client can fix first.

TL;DR

A captcha is a policy action on a risk score. reCAPTCHA v3 scores each request from 0.0 to 1.0 with 0.5 as a suggested starting threshold, and each site owner decides what happens below it.
The proxy supplies the IP inputs. ASN class, the shared history of the address and how many requests it carries all come from the proxy tier.
The client supplies the rest. Pace, cookies, the TLS stack, the browser checks and the locale belong to the client on every tier.
Per-request rotation can raise the challenge rate. Each new address starts without the session history the last one built, so multi-page flows often do better on a sticky session.
Some captchas mean stop. When a site challenges every automated request on logins, checkout or search, the answer is its API, a data licence or slower permitted access, not another pool.

Key takeaways

Measure before switching tiers. Put reCAPTCHA v3 or Turnstile on a page on a domain you own, record 100 requests per tier at a fixed pace, then change one client input and repeat.
Keep clearance cookies. Cloudflare's cf_clearance cookie lasts 30 minutes by default and Cloudflare recommends 15 to 45; a client that drops it meets the next challenge sooner.
Use sticky sessions for multi-page flows. A 5 to 60 minute timer or a sticky session keeps the history that per-request rotation throws away.
Ask for the market's language. Set Accept-Language to the market whose pages the job needs, so the site serves the right locale.
Use mobile where the mobile origin matters. Carrier IPs are built for platforms that expect phone traffic; residential covers most other targets.

Read the full guide on proxymint, with the commands, tables and sources.

USA residential proxies: state, city and session guide

Andrew — Thu, 08 Oct 2026 12:05:40 +0000

Originally published on proxymint.

USA residential proxies route through US home connections. How state and city targeting works, which session type fits, and how to check the exit.

TL;DR

A US residential proxy exits from a US home connection. The address belongs to a consumer ISP network, so databases classify it as residential and place it in the US, usually in a state and city too.
State is the reliable level in the US. MaxMind places about 80% of US addresses in the right state but 66% within 50 km of the right city, and the state carries tax, licensing and regional promotions.
Rotation follows the flow. Per-request rotation suits stateless pages, a 5 to 60 minute timer suits short cookie flows, and sticky holds while the home device stays online.
The IP is one US signal of several. Time zone, locale and the DNS path have to match the exit, and Arizona and Hawaii skip daylight saving time.

Key takeaways

Pick the level from the page. Use country for national pages, state when prices, tax or licensing change by state, and city only for local search, store stock or metro ads.
Check exits in two databases first. Echo the IP, confirm a consumer ISP network and the state in two lookups, and confirm DNS resolves on the proxy side.
Set time zone and locale per state. Run each US market in a fresh browser context with en-US and the IANA zone of the exit city.
Set ZIP inside the site. A city exit plus the site's own store or ZIP picker reaches a ZIP-level page without ZIP targeting.
Use static ISP for a long-lived US identity. A home device can go offline; one assigned US ISP address holds for the whole term.

Read the full guide on proxymint, with the commands, tables and sources.

Datacenter vs residential proxies: which fits the target

Andrew — Thu, 08 Oct 2026 08:05:43 +0000

Originally published on proxymint.

Datacenter vs residential proxies compared on ASN, speed, location and billing model, with a 100,000-request job sized in GB and a test to pick per target.

TL;DR

The ASN is the difference the target sees first. A datacenter IP is announced by a hosting company, often from published ranges; a residential IP is announced by a consumer ISP on a real home connection.
Test the target before picking a tier. A sample of a few hundred requests at the real pace shows whether the target serves normally, blocks, challenges or serves different content.
Residential is usage-based, so page weight sets the traffic. 100,000 HTML-only requests at 22 KB move 2.2 GB; the same requests as full 2,164 KB renders move 216.4 GB.
Datacenter fits repeating or heavy jobs on lenient targets. A dedicated IPv4 port is billed per month with no bandwidth cap under normal use, so pace sets the limit rather than GB.
Datacenter is usually faster, but measure it. Server uplinks beat home lines on latency, while residential latency varies by exit; compare p50 and p95 time to first byte on the real target.

Key takeaways

Sample every new target on datacenter first. Sort the responses into serves normally, blocks, challenges or serves different content, and let that decide the tier.
Spot-check datacenter data against a home connection. A 200 status with different prices or rankings is silent bad data, and it is the one failure that looks like success.
Count valid records, not requests. Only responses whose content matches the home control count toward the job.
Rule out the client before blaming the proxy. Run the same requests at the same pace from a home network with no proxy; if they are challenged there too, a better IP will not help.
Route per target, not per company. Datacenter for lenient targets, residential for targets that block hosting ranges, static ISP when one address must hold for days, mobile only where a mobile origin is required.

Read the full guide on proxymint, with the commands, tables and sources.

Do proxy providers keep logs? What they record and why

Andrew — Wed, 07 Oct 2026 18:05:42 +0000

Originally published on proxymint.

TL;DR

Every proxy provider keeps some logs. Billing per GB, handling abuse reports and answering legal process all need records, so the real question is which records and for how long.
Over HTTPS a proxy sees the destination, not the content. A CONNECT tunnel names only the host and port; paths, headers, passwords and page content stay inside TLS.
"No logs" claims have failed under court orders. Records from a web proxy service in 2011 and a VPN with a no-logs policy in 2017 both ended up in criminal cases.
A good policy names its records. Look for the fields logged, the purpose, the rule that sets retention, the disclosure rule and your rights.

Key takeaways

Read the privacy page before buying. Check for named records, stated purpose, a retention rule, a legal-request rule, no data resale and a way to request your data.
Send traffic only to HTTPS targets. Plain HTTP requests reach the proxy with the full URL, headers and body readable.
Never install a provider root certificate. Leave certificate checks on; a trusted proxy certificate is the only way a proxy can read HTTPS content.
Choose where DNS resolves on purpose. socks5h sends host names to the proxy; socks5 resolves them locally where your own resolver sees them.

Read the full guide on proxymint, with the commands, tables and sources.

Proxy format: IP, port, username and password, every form

Andrew — Wed, 07 Oct 2026 13:28:42 +0000

Originally published on proxymint.

Read and write proxy strings: host:port:user:pass, user:pass@host:port, http:// vs socks5h:// URLs, separate fields, password encoding and IPv6 brackets.

TL;DR

Four forms carry the same four values. The colon list host:port:user:pass, the authority user:pass@host:port, the full URL with a scheme, and separate server, username and password fields.
The full URL is the portable form. curl, Python requests, Node agents, Scrapy and environment variables all parse scheme://user:pass@host:port, and the scheme picks HTTP, SOCKS5 or SOCKS5 with proxy-side DNS.
Special characters need percent-encoding inside a URL. Write @ as %40, : as %3A, / as %2F and # as %23; separate fields take the raw password.
IPv6 hosts go in brackets. http://user:pass@[2001:db8::1]:8080 is the only safe way to write one; the colon list cannot carry IPv6.

Key takeaways

Store each proxy once as a full URL. Keep the password percent-encoded and generate colon lists or separate fields from that one source.
Encode credentials separately, never the whole URL. Use quote(value, safe='') in Python and encodeURIComponent in JavaScript.
Use socks5h:// or an http:// proxy when location matters. Plain socks5:// resolves the target name on the client's own machine.
Give Playwright and Puppeteer separate fields. Playwright drops credentials written inside server and refuses SOCKS5 with a username, so authenticated browsers use the HTTP endpoint.
Read the error before changing the string. A parse error means a malformed string, a 407 means wrong credentials, and a refused connection means the wrong host, port or scheme.

Read the full guide on proxymint, with the commands, tables and sources.