DEV Community: Maxi

3.6 million checks of free proxy lists in a day: what the numbers show, and what they don't

Maxi — Fri, 16 Oct 2026 21:01:57 +0000

Free proxy lists have a reputation: most entries are dead, and the ones that answer are slow, blocked or worse. I wanted numbers instead of a reputation, so proxy-scraper checks the public lists about every hour in GitHub Actions and publishes what passes.

This post has the numbers from 24 of those runs (28 Sep 09:43 UTC to 29 Sep 08:39 UTC, 2026) and from a week of published lists. An earlier version of it drew conclusions the data doesn't carry. This one says for every number what was counted, out of what, and what it can't tell you. Every number in the two main sections comes out of a script in docs/analysis/ that you can run yourself.

The funnel

Each run reads about 800 public proxy lists and ends up with 1.26 million candidates on average. A candidate is a protocol plus an address: the same ip:port listed as HTTP and as SOCKS5 counts twice. The run then checks the 150,000 most promising ones. Proxies that passed before and lists with a good hit rate go first.

24 runs × 150,000 = 3.6 million checks. That is not 3.6 million proxies. The same sources go first in every run, so the 24 samples overlap, probably heavily. How many distinct candidates were checked that day was never recorded.

Step	What has to happen	Over 24 runs	Share
First request	the proxy speaks HTTP, SOCKS4 or SOCKS5 and returns a public exit IP from a check target	93,244 answered	2.6 % of checks
Second request	a request to a second site (httpbin.org) returns valid JSON with the same exit IP	50,193 did not pass	54 % of those that answered (41–67 % per run)
Content	a static test page arrives byte for byte	2,244 arrived with a different hash	at least 5.2 %: 2,244 of the 43,051 that got this far
Kept	the first two passed and the test page was not seen to differ	40,807	1.1 % of checks

python docs/analysis/funnel_summary.py docs/analysis/data/runs-2026-09-28_29.csv 2026-09-28T09:40 2026-09-29T08:45

The per-run numbers in that CSV were read from the workflow's console logs (funnel_from_logs.py). GitHub deletes those logs after 90 days, so the CSV is the copy that stays.

What the funnel doesn't tell you

Why the second request failed. The logs have a count and no reasons. A proxy fails this step if it can't load two pages in a row, if its exit IP changes between the two requests (a rotating or chained exit), if it only answers the first check, or if the second site turns it away. The earlier version of this page called these "honeypots". Some may be. Nothing that was measured says how many.

It also counts httpbin.org's own bad minutes. Until 1.30 the tool asked httpbin.org directly once, at the start of a run, and never again. If it went down halfway, every proxy after that failed the second request. None of the 24 runs logged a missing second target at the start; what happened during them is not in the logs.

Why the page differed. The check compares one hash. A page cut off by the proxy, a cached older copy, a block page with status 200 and an injected script all give a different hash. The earlier version said these proxies "usually inject a script". That came from one early look at 270 proxies, of which 54 returned a changed page. No sample of those pages was kept, so it is not a finding. The earlier version also gave the share as "1 in 40 working proxies", which used the wrong denominator: 2,244 is 2.4 % of everything that answered, and 5.2 % of the proxies that reached this check.

How many pages were compared. In these runs a content request that timed out or came back as an error page counted as "unchanged", and the proxy was kept. So the 40,807 include an unknown number of proxies whose test page never arrived, and 5.2 % is a lower bound for the share of compared pages that differed.

How many free proxies work. 1.1 % is the share of a prioritised sample. Known-good proxies go first, so a random entry of these lists does worse. The checks ran with a 3 s connect timeout and 6 s overall, from GitHub-hosted runners. A proxy that fails from there can work from somewhere else.

What a "not checked" proxy looks like. Of 1.26 million candidates per run, 1.11 million were never checked. The runs recorded nothing about them. They are not "dead", they are unknown.

How long entries stay on the list

Every published run notes which entries are on the list. seen.json keeps that as one bit per entry and run for the last week. The numbers below are from the week ending 13 Oct 2026, 16:47 UTC (156 published runs).

python docs/analysis/list_churn.py list/        # see the top of the script for how to get the three files

First, what an entry is. The week saw 41,634 different type://ip:port entries. They sit on 4,890 hosts. 82 % of all entries are ports on 16 hosts, each with 50 or more entries ("crowded hosts" below). The list on 13 Oct had 3,074 entries on 527 hosts with 618 different exit IPs. So "3,000 proxies" is closer to 500 machines, and every statistic about entries is mostly a statistic about a few hosts with hundreds of ports whose entries come and go from the list.

	per entry	per host
On the list at least once	41,634	4,890
Off the list at the end of the week	38,398	4,333
… whose last stretch on the list was a single run	32,819 (85 %)	3,666 (85 %)
… whose last stretch was 24 runs or more	12	12
Every stretch that began and ended inside the week	142,517	36,034
… one run long	111,059 (78 %)	26,742 (74 %)
… 24 runs or more	215	203
Still listed, for 24 or more runs so far	57	71

The earlier version of this page said of the entries that had dropped off the list: "85 % were gone by the very next check, an hour later", and concluded that a day-old list is mostly dead. The 85 % is real, but it is narrower than it sounds: it counts only entries that are off the list at the end, and only the last stretch of each. Counting every stretch, 78 % were one run long, and 215 lasted a day or more, not 12. Entries that stay are in neither number. Asked the other way round:

Of the entries on one run's list …	all entries	without the 16 crowded hosts
… on the list again one run later	40 % (16–58 %)	57 % (20–84 %)
… on the list again 24 runs later	26 % (8–40 %)	49 % (17–69 %)

Median over the week's runs, range in brackets. So about a quarter of a list is on the list again a day later, and about half once the 16 crowded hosts are left out. The earlier sentence "a list that is a day old is mostly a list of dead proxies" doesn't hold for the second group.

What this doesn't tell you

That an entry failed. Not being on the list is not a recorded failure. An entry is missing from a run if it failed a check, but also if the run didn't get to it, if the runner was slow that hour, or if the second check target had a bad moment. Before 1.30 the published data can't tell these apart.

How long a proxy lives. Runs are about hourly: the median gap was 0.96 h, eight gaps were over 1.5 h and one was 13.9 h. "24 runs" is roughly a day, not exactly. Stretches that began before the week or haven't ended are left out of the "left the list" rows, and those are the long ones.

One list, looked at closely

These are single snapshots. They are here because they were in the first version, with what they are worth.

Big sites. After each run the HTTPS-capable entries send one request each to a few big sites. The table shows the share that got a normal page, out of those that gave a clear answer, for three stored lists:

Site	29 Sep, 01:14	13 Oct, 01:23	13 Oct, 16:47
Google	186 of 776 (24 %)	594 of 716 (83 %)	611 of 916 (67 %)
TikTok	401 of 734 (55 %)	394 of 628 (63 %)	652 of 813 (80 %)
Reddit	345 of 766 (45 %)	214 of 572 (37 %)	243 of 867 (28 %)
Instagram	323 of 801 (40 %)	106 of 805 (13 %)	153 of 1,103 (14 %)
Amazon	242 of 856 (28 %)	22 of 795 (3 %)	226 of 1,169 (19 %)

The first version showed one run (Google 47 %, Amazon 2 %) and concluded "Amazon blocks almost everything". The three columns above differ by a factor of up to ten for the same site, so one run is not a ranking of sites. It mostly reflects which gateways were on the list that hour. The split into "datacenter" and "other" exits in the first version rested on whether a provider's name looks like a hosting company. "Other" is everything else. It was never shown to be home or mobile lines.

What the entries on a list look like (list of 29 Sep, 08:49 UTC, 2,914 entries, from the run's log and history.json):

62 % have a provider name that looks like a hosting company; in the three lists above it was 48 %, 62 % and 65 %
28 % exit from an IP on the SpamCop blocklist; 43 %, 36 % and 34 % in the three lists above
64 % passed the HTTPS test: a tunnel to one test host with a certificate that verifies, at check time
median first answer after 1.9 s; median download speed 68 KiB/s among the HTTPS-capable entries from which at least 20 KB of a 100 KB download arrived within 12 seconds
most entries were SOCKS5, and most SOCKS5 entries are ports on the crowded hosts (937 of 1,122 in that day's stored list, the 01:14 snapshot, all in one /24), so this says little about the protocols

What changes with 1.30

The gaps above are gaps in what was recorded. From 1.30 on:

every check of every kept proxy ends as passed, failed, unavailable or skipped, with a reason code, the target and the time (check states)
a check target that goes down mid-run no longer fails the proxies checked against it; they are marked unavailable
every run writes run.json: candidates, checked, not checked, and each check by state and reason. The hourly list publishes it and keeps the counts in history.json
run.json counts how many of the changed test pages contain a <script tag and how many don't (the reference page has none), and --diagnose shows size and hash of the page one proxy returned. That is still not a stored sample, but it is the first thing that could support or refute "scripts"

A follow-up with a few weeks of that data can answer what this page can't: how the second request fails, and how often the page differs for a reason that matters.

Try it

The recording runs against six simulated proxies on localhost. It shows how results are reported and measures nothing about public proxies.

The current list with filters: https://maximilianfeix.github.io/proxy-scraper/
The lists alone, per protocol and country: https://github.com/maximilianfeix/free-proxy-list
Check one proxy and see what each check found: uvx proxy-scraper-cli --diagnose socks5://HOST:PORT
Source, MIT licensed: https://github.com/maximilianfeix/proxy-scraper

Free proxies are run by strangers. Never send passwords, cookies or personal data through them.

Disclosure: this post was drafted with an AI assistant from the repository's own analysis scripts and data; every number comes from the scripts linked above. The canonical version lives in the repository.

What changed in this Electron update? Diffing two app.asar builds without running either

Maxi — Fri, 16 Oct 2026 20:52:39 +0000

Your Electron app's 1.5.0 is ready. Can a web page in it now reach something it couldn't in 1.4.0? Did a window lose its sandbox? Does the code call a server it didn't call before?

The diff of two minified app.asar bundles won't tell you. Running the new build to find out is exactly what you wanted to avoid. So I built a tool that reads both builds statically and tells you what changed, with the file and line behind every change, and an honest list of what it couldn't figure out.

Everything below is reproducible in about ten seconds. You need Node 22+.

1. Two builds, one command

npx hexvero demo

This copies two builds of a small example Electron app (NotesDesk 1.4.0 and 1.5.0, written for these examples, nothing from a real product) into ./hexvero-demo and compares them.

2. What a page can reach

In Electron, a page talks to the main process through the preload script: contextBridge exposes members, each sends on an IPC channel, and ipcMain.handle answers. hexvero joins the three:

interfaces (2)
  + preload.js::notesdesk::checkUpdate  @ b: preload.js:9, handler main.js:30 · link direct
  - preload.js::notesdesk::openInShell  @ a: preload.js:9, handler main.js:30 · link direct

1.4.0 let any page open a path in the system shell through openInShell. 1.5.0 removes it and adds checkUpdate, whose handler at main.js:30 fetches an update URL. link direct means the bridge member and the handler use the same literal channel, so the link isn't a guess.

3. What a window may do

windows (1)
  ~ main.js::win  @ b: main.js:8
      nodeIntegration: true → false, nodeIntegrationInEffect: "on" → "off", sandbox: false → true

Settings are reported as written. If a value isn't written and rests on Electron's default for the version in package.json, it says default on / default off instead of pretending it read it.

4. What it could not tell

This is the part I care about most:

Not established
  ! b: 1 IPC call(s) pass a channel that is not a literal or a constant that could be followed
    (a parameter, a computed value). … ipcMain.handle at main.js:33. (ipc_channel_unresolved)

1.5.0 registers plugin:${name} channels with names from an environment variable. No static reading can list them. So instead of guessing (or silently dropping them), hexvero names the call and the line, and the IPC finding turns from "confirmed" to "inferred". Same in comparisons: an item in a file the other build didn't read is unknown, never removed.

5. Check it, write it down

npx hexvero ui --demo --open
npx hexvero compare hexvero-demo/notesdesk-1.4.0.asar hexvero-demo/notesdesk-1.5.0.asar --report report.md

The local web UI (127.0.0.1, per-session token, nothing loaded from the network) opens the source lines behind every file:line. The report is plain Markdown you can attach to a release review; same inputs, same bytes. Here's the example report.

What it doesn't do (honestly)

It reads files; it never runs the app. A finding is what the code contains, not what it does, and not proof that anything is exploitable.
It misses things. On my newest blind test set (an app written by someone without access to hexvero, measured once) it found 12 of 57 IPC channels and 5 of 8 version changes, mostly because channels came from a registry function it doesn't evaluate. The misses are published in the repo.
"Doesn't execute the target" isn't a safety guarantee: parsing hostile bytes is still parsing hostile bytes (bounded in size, depth and time).

On your own app

npx hexvero compare old/MyApp.app/Contents/Resources/app.asar \
                    new/MyApp.app/Contents/Resources/app.asar --report report.md

It also reads ELF/PE/Mach-O, APKs, .NET, WebAssembly and more, and ships an MCP server so a coding agent (Claude Code, Cursor, Codex…) can run the same investigation with small, cited answers.

It's MIT and open source: github.com/maximilianfeix/hexvero. If it gets a fact wrong on your app, an issue with the file and line is the most useful thing you can send me. And if it showed you something you didn't know about your build, a star helps others find it.

I'm the author of hexvero. This post was written by an AI agent working on my behalf; every output in it was produced by running the tool. Only use it on software you own or are allowed to examine.

docker builder prune freed 0 B: two places Docker keeps disk space that no prune reaches

Maxi — Thu, 15 Oct 2026 05:33:22 +0000

Disclosure: this article was drafted with AI assistance (Claude) from measurements recorded in the dockdu repository. Every number comes from the measurement log and can be reproduced on a disposable docker:dind container.

docker system df says the build cache is empty. docker system prune -af reports Total reclaimed space: 0B. df -h still shows the disk filling up. Here are two cases where all three statements are true at once, both measured on Docker Engine 29.9.0, with the commands to check each by hand.

1. The build cache of a buildx builder is a volume

docker buildx create --driver docker-container starts BuildKit in its own container, buildx_buildkit_<node>. The builder keeps its whole state, including its build cache, in a volume named buildx_buildkit_<node>_state (buildx source). The node is usually the builder name plus 0.

Measured: one builder ci, one build that added 30 MB.

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          2         1         430.9MB   73.34MB (17%)
Containers      1         1         40.96kB   0B (0%)
Local Volumes   2         1         73.31MB   114.7kB (0%)
Build Cache     0         0         0B        0B

The 73 MB of build cache is in the Local Volumes row. Then the two prune commands, with du -sk on the state volume after each:

Command	State volume on disk	Prune printed
(before)	72452 KiB
`docker builder prune -af`	72452 KiB	`Total: 0B`
`docker buildx prune --builder=ci -af`	740 KiB	`Total: 73.36MB`

docker builder prune works on the selected builder, which here was the daemon's own. A CI host that once ran docker buildx create --use and was later switched back keeps growing in a volume that docker builder prune never touches.

There is one more trap. docker buildx rm --keep-state removes the builder but keeps the volume. Nothing then refers to it, so it only shows up as an unused volume.

Check by hand:

docker volume ls --filter name=buildx_buildkit_      # one per docker-container node
docker system df -v | grep buildx_buildkit_          # sizes
docker buildx ls                                     # which builder each node belongs to
docker buildx du --builder <name>                    # that builder's cache, record by record

Free it: docker buildx prune --builder <name>. Add --keep-storage 10gb to keep a working set. Or set a GC policy in that builder's buildkitd.toml.

2. Switching to the containerd image store hides your old images

Docker Desktop and fresh Engine 29 installs use the containerd image store. If a daemon is switched to it, the images and containers of the classic overlay2 store are not deleted. Docker's docs say they become hidden: "Your overlay2 data remains on disk but becomes hidden."

Measured: one data root with alpine:3.22 and nginx:1.29-alpine pulled under the graph driver, then the same data root started with --feature containerd-snapshotter=true:

$ docker images                 # (empty)
$ docker system df              # every row 0B
$ docker system prune -af
Total reclaimed space: 0B
$ sudo du -sh /var/lib/docker/overlay2
71.8M   /var/lib/docker/overlay2

On a real machine that number is whatever you had before the switch, often many gigabytes.

Check by hand:

docker info --format '{{.Driver}} {{json .DriverStatus}}'    # io.containerd.snapshotter.v1 = containerd store
sudo ls /var/lib/docker/image/overlay2/layerdb/sha256 | wc -l  # layers the classic store still records
sudo du -sh /var/lib/docker/overlay2

Free it: switch the daemon back to overlay2. The old images and containers reappear and can be removed normally. docker save anything you want to keep first.

What does not hide space: containerd's compressed content

A common suspicion is that the containerd store's second copy of each layer (compressed blobs next to unpacked snapshots) is space df ignores. It is not, at least on 29.9:

$ docker system df     → Images 296.5MB
content (compressed)     72.4 MB
snapshots (unpacked)    224.4 MB      sum 296.8 MB

The Images row includes both.

Leftover overlay2 directories

The third classic case, moby#47089: overlay2/ is far larger than images, containers and build cache together. The graph driver records which directory belongs to which image layer (image/overlay2/layerdb/sha256/*/cache-id) and to which container (layerdb/mounts/*/mount-id). A directory that appears in neither is a BuildKit snapshot or a leftover. Comparing the size of those directories with the private build cache shows whether leftovers are the likely cause. Don't delete them by hand while the daemon runs.

Tooling

dockdu, a read-only CLI that explains Docker disk usage, does all three checks since 0.4.0:

buildx builders are listed under BUILD CACHE, with the prune command for each;
the hidden classic store is reported from a single directory listing;
dockdu --disk splits overlay2/ into image layers, container layers and directories with no record.

It never deletes anything, and its Docker client refuses any request other than GET/HEAD on four read endpoints. The manual commands above work without it.

Why docker system df doesn't explain your full disk, and why prune frees less than it says

Maxi — Wed, 14 Oct 2026 19:47:19 +0000

Disclosure: this article was generated by an AI agent (Claude) working on the dockdu repository, from measurements it ran on disposable Docker daemons. Every number links to its raw output and can be reproduced with the commands below.

Measured on 2026-10-14 on disposable Docker Engine 28.5.2 and 29.9.0 daemons. Every number below is from the measurement log; the commands work on any Linux host where you can read /var/lib/docker (as root).

docker system df is precise about what it counts. The trouble is what it leaves out, and that "reclaimable" means two different things depending on your Docker version and image store. This article goes through the four places the gap usually hides, with commands that answer each question by hand.

1. Container logs are not in any `df` row

The default log driver, json-file, has max-size unlimited and max-file 1 unless you configure it (docs). The log of a chatty container grows until the disk is full. docker system df has four rows (Images, Containers, Local Volumes, Build Cache), and log files are in none of them. Containers' SIZE is the writable layer only. Adding logs to df has been requested since 2017 (moby#32205, moby#38341).

Measured: one container wrote 500 MiB to stdout.

$ docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          2         2         147.7MB   0B (0%)
Containers      4         4         24.58kB   0B (0%)
Local Volumes   2         1         33.55MB   0B (0%)
Build Cache     4         0         147.6MB   12.29kB

$ du -ch /var/lib/docker/containers/*/*-json.log | tail -1
535.1M  total

The logs are larger than everything df reports combined. docker system prune does not touch them: in the same setup it reclaimed 0B, because the containers were running.

Check by hand:

sudo du -h /var/lib/docker/containers/*/*-json.log* | sort -h | tail
docker inspect -f '{{.Name}} {{.HostConfig.LogConfig.Type}} {{.HostConfig.LogConfig.Config}}' $(docker ps -aq)

Fix:

Set "log-opts": {"max-size": "10m", "max-file": "3"} in /etc/docker/daemon.json, or use the local driver (20 MiB × 5 files, compressed by default).
Both apply to new containers only; existing ones must be recreated.

Edge cases:

Rotated files (-json.log.1, .2) count too.
Truncating vs deleting: deleting a log file of a running container does not free the space until the daemon closes it. A deleted file that is still open shows up in lsof +L1, not in du. Truncating it to zero frees it at once but discards history.
Other drivers: with journald or syslog the logs live outside Docker's data root.

2. Build cache "shared" with images: counted twice, freed differently

BuildKit marks a cache record Shared when its data is also part of an image. Three tools give three different answers for the same records:

	Engine 28.5.2, overlay2	Engine 29.9.0, containerd image store
`docker system df` Build Cache row	leaves shared records out (96 B shown, 64 MiB shared)	includes them (147.6 MB)
`docker buildx du` Reclaimable	counts shared records as reclaimable	counts shared records as reclaimable
`docker builder prune -af` prints	`Total: 67.11MB`	`Total: 147.6MB`
Freed on disk (`du -sxk` before/after)	≈ 0	72.7 MiB of 140.7 MiB

Why the rows differ:

Docker CLI 28 sums only records that are not shared (cli/command/system/df.go, v28.5.1).
Engine 29 (API ≥ 1.52) sends a total that includes them.

Why prune prints a number it does not free:

On overlay2, a shared record's data belongs to the image, so pruning the record removes metadata.
On the containerd image store, about half of the shared bytes were freed in two separate runs (72.7 of 140.7 MiB, 56.8 of 108.7 MiB). The build cache evidently kept its own copy of part of the data; these measurements do not show which store objects it consisted of.

What this means in practice:

On overlay2, expect docker builder prune to free roughly the private part only.
On the containerd image store, expect somewhere between the private part and the printed total. Measure with du before and after if it matters.
When adding up df rows, Engine 29 counts shared bytes in both the Images and the Build Cache rows.

Check by hand:

docker buildx du --verbose | grep -E 'ID:|Shared:|Size:'      # per record
docker info --format '{{.Driver}} {{json .DriverStatus}}'     # "io.containerd.snapshotter.v1" = containerd image store
sudo du -sxk /var/lib/docker      # before and after a prune: what it really freed

Fresh Engine 29.9.0 installs use the containerd image store by default (observed in docker:29-dind), and so does Docker Desktop. Hosts upgraded from older versions may still be on overlay2.

3. Image sizes overlap

docker image ls shows each image's full size. Two images built FROM the same base each include that base, so adding up image sizes, or per-project totals built from them, counts the base more than once. The Images row of docker system df counts each layer once.

In the demo the two images share 12.7 MiB. docker system df -v shows SHARED SIZE and UNIQUE SIZE per image; only the unique size comes back when you remove an image that no container uses.

4. Docker Desktop: a sparse file

On macOS, all of Docker lives in one VM disk file.

Finder and ls -l show its maximum size; ls -ks shows what is really allocated (Docker Mac FAQ).
Freeing space inside the VM does not shrink the file until Docker Desktop compacts it.
Logs are invisible from the host: the log files from section 1 live inside the VM, where a host-side du cannot reach them. Run commands inside the VM instead, for example with docker run --rm -v /var/lib/docker:/var/lib/docker:ro alpine du ….

ls -ks ~/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw

When none of this adds up: is it Docker's at all?

Compare the filesystem with the data root, both in allocated blocks on the same filesystem:

df -h /var/lib/docker
sudo du -sxh /var/lib/docker          # -x: do not count overlay mounts of running containers
sudo du -sxh /var/lib/docker/*/ | sort -h

What the result tells you:

The data root is small: the full disk is not Docker's.
overlay2/ or containerd/ is far larger than images + containers + build cache: look for leftovers from crashes or interrupted pulls (moby#47089, moby#52539).

Don't subtract docker system df numbers from du numbers: one is logical sizes with shared layers counted once, the other is blocks on disk.

The one-command version

dockdu, an open-source tool by the author of this account, does the checks above in one read-only run. It:

measures log files (with lstat only);
splits the build cache into shared and private and says what prune keeps, or that it is unknown on the containerd image store;
groups everything by compose project;
with --disk, measures the data root next to the filesystem.

Its Docker client refuses any request other than GET/HEAD on four read endpoints, and it never deletes anything.

brew install maximilianfeix/tap/dockdu      # or the container image, see the README

It does not attribute disk space that no API reports (overlay2 leftovers), it only counts the default BuildKit builder, and it does not run on Windows. The manual commands above work everywhere.

Reproduce every number: scripts/demo-up.sh on a disposable daemon, then the commands in the measurement log.

I checked a million free proxies. One in five of the working ones rewrote my pages.

Maxi — Sat, 26 Sep 2026 09:47:46 +0000

Free proxy lists look generous. Thousands of addresses, updated every few minutes. The first time I tried them for a scraping job, almost nothing connected. The second surprise came later: some of the ones that did connect were not doing what I thought.

This is what I found while building proxy-scraper, an open-source tool that collects public proxies and checks them properly.

Dead is the easy part

Every run starts with about a million candidates from 700+ public lists. Most of them never finish a TCP handshake. That's expected and cheap to detect.

The interesting failures come after that.

Honeypots answer the question you asked

The usual check is: connect through the proxy, ask a "what is my IP" service, see a foreign IP, done. Some proxies answer exactly that request with a plausible "200 + IP" and fail everything else. In some runs they were five out of six "hits".

The fix is a second, independent request. The proxy has to fetch a different site and show the same exit IP there too. Honeypots fall out here, and the same response tells you which headers arrive at the target, which gives you the anonymity level for free.

Some proxies edit what you download

This is the one that surprised me. I added a check that fetches a static HTML page through the proxy and compares its hash with the page fetched directly over verified HTTPS.

Out of 270 proxies that had passed the handshake and the honeypot check, 54 returned a modified page. Almost always an injected <script src="http://…">, sometimes ads. That's 20% of the pool that looked perfectly fine by every usual measure.

If you scrape through free proxies, a fifth of your responses may be rewritten on the way. If you browse through them, you're running someone else's JavaScript.

Where the rest exits from

With the free DB-IP ASN database the tool looks up the provider of every exit IP. In today's list, 52% of working proxies exit from datacenters, which many sites block on sight. And 28% of exit IPs are on the SpamCop blocklist, which means captchas on a lot of sites. Both can be filtered out.

Learning which lists are worth it

A proxy that worked in the last run is far more likely to work now than a random entry. So proxy-scraper checks known-good proxies first, then proxies from sources with a good hit rate, and it drops lists that went stale. Unchanged lists are skipped with ETags: a second run right after the first loads 0 MB instead of about 160 MB.

Try it

pipx install git+https://github.com/maximilianfeix/proxy-scraper.git
proxy-scraper --want 50 --https-only

--serve turns the result into a rotating proxy (HTTP and SOCKS5 on one port, sticky sessions, country per request through the username, optional password, refills itself in the background), there's a Python API, and -o - prints the hits for pipes. If you don't want to run anything, a GitHub Action checks everything every hour and publishes the list: maximilianfeix.github.io/proxy-scraper

The code is MIT licensed. If you find a check I'm missing, issues are very welcome: github.com/maximilianfeix/proxy-scraper

Free proxies are run by strangers. Never send passwords or personal data through them.

DEV Community: Maxi

3.6 million checks of free proxy lists in a day: what the numbers show, and what they don't

The funnel

What the funnel doesn't tell you

How long entries stay on the list

What this doesn't tell you

One list, looked at closely

What changes with 1.30

Try it

What changed in this Electron update? Diffing two app.asar builds without running either

1. Two builds, one command

2. What a page can reach

3. What a window may do

4. What it could not tell

5. Check it, write it down

What it doesn't do (honestly)

On your own app

docker builder prune freed 0 B: two places Docker keeps disk space that no prune reaches

1. The build cache of a buildx builder is a volume

2. Switching to the containerd image store hides your old images

What does not hide space: containerd's compressed content

Leftover overlay2 directories

Tooling

Why docker system df doesn't explain your full disk, and why prune frees less than it says

1. Container logs are not in any df row

2. Build cache "shared" with images: counted twice, freed differently

3. Image sizes overlap

4. Docker Desktop: a sparse file

When none of this adds up: is it Docker's at all?

The one-command version

I checked a million free proxies. One in five of the working ones rewrote my pages.

Dead is the easy part

Honeypots answer the question you asked

Some proxies edit what you download

Where the rest exits from

Learning which lists are worth it

Try it

1. Container logs are not in any `df` row