I measured how many pages fit in 1 GB of proxy traffic

Lucas — Wed, 07 Oct 2026 10:01:58 +0000

Proxy plans are sold per GB, but scrapers think in pages. So when someone asks "how many GB do I need?", the usual answer is "it depends". That's true, and it doesn't help anyone pick a plan.

So I measured it. 34 public pages, fetched three ways, with every byte counted at the tunnel in both directions.

The short answer

How you fetch	Median per page	Pages per GB
HTTP client, HTML only	22.7 KB	44,083
Playwright, full load	1.10 MB	906
Playwright, images/media/fonts blocked	454.6 KB	2,199

Same pages, same day. At the median, the full browser load used about 49 times the traffic of a plain HTML fetch.

How I measured

34 public pages: docs, news articles, blog posts, forum threads, shop category and product pages, and a few JavaScript apps. Only pages robots.txt allows, nothing behind a login.
Three modes: Requests 2.34.2 (one GET, compression on), Playwright 1.63.0 full load, and Playwright with images, media and fonts blocked.
Three runs per page, fresh process and empty cache every time. 306 loads, all HTTP 200.
All traffic went through scrapescope, a small local proxy I built that counts bytes per tunnel. So the numbers include TLS, headers and upload, not just page weight.
October 5, 2026, from Helsinki, direct connection.

Three things that surprised me

1. Scripts, not images. In full loads, scripts were 49% of the bytes. Images, media and fonts together were 35%. Blocking them saved 28% at the median, less than I expected. On news pages the blocked loads were actually heavier (2.24 MB vs 1.32 MB median), because pages behave differently when things fail to load.

2. Half the bytes go to other companies. 49.9% of full-load bytes went to third-party hosts. Google Tag Manager alone was on 17 of the 34 pages at about 308 KB per load. That's 12% of all full-load bytes, for analytics your scraper will never use.

3. Compression does a lot of quiet work. All 34 HTML responses were compressed (17 Brotli, 14 gzip, 3 zstd). The median HTML document was 83.5 KB decoded but 22.7 KB on the wire. A client that doesn't ask for compression pays for the bigger number.

Upload counts too

Upstream was 6.3% of the traffic for HTML-only fetches and about 3% for browser loads. Small, but if your provider bills both directions, it's on your bill.

Estimate your own

GB per day = pages per day × bytes per page × attempts per page ÷ 1,000,000,000

50,000 pages a day with a full browser (1.10 MB median, 1.2 attempts per page) comes to about 66 GB a day. The same job with an HTTP client is about 1.4 GB.

Your pages won't match mine, so treat these as a starting point. Better still, run your own job through scrapescope for a day and use your numbers.

Limits

One location, one day, 34 pages. Direct connection, so no proxy exit location, blocks or retries. CONNECT bytes are estimated. Pages change.

The full write-up has the breakdown by page type, cost per 1,000 pages at different per-GB prices, the raw CSV of all 306 runs and the scripts: How many GB of proxies do you need?

scrapescope is open source: github.com/ipvolt/scrapescope

Disclosure: I'm building ipvolt, a proxy service that bills per GB, which is why I wanted real numbers. scrapescope works with any provider.

What do your scrapers average per page? I'm curious whether anyone sees numbers very different from these.

My coding agent kept "fixing" proxy errors wrong, so I built an MCP server

Lucas — Fri, 02 Oct 2026 13:08:43 +0000

I work on proxy infrastructure, which means I read a lot of proxy errors. And I kept noticing the same thing: the coding agents I used were great at almost everything except this.

Give one a 407 and it would happily "fix" it by putting the proxy credentials into the target site's Authorization header. Show it a timeout on a POST and it would just retry, without asking whether the first request had already reached the server.

That's not the model being dumb. Proxy errors are a narrow area with a lot of confidently wrong answers floating around the internet, and the model has read plenty of them.

So I built a small MCP server that gives the agent grounded answers for this one area. It's free, open source and works with any proxy provider. Here's what I learned designing it.

1. Don't let the agent paste raw logs

The diagnose tool doesn't accept logs, URLs or credentials. It takes structured observations instead. This is a real input for the 407 case:

{
  "client": "requests",
  "version": "2.34.2",
  "phase": "connect_tunnel",
  "status": 407,
  "responseSource": "proxy"
}

Two reasons. Logs are full of things that shouldn't end up in a model's context, like credentials and session cookies. And the phase matters more than people expect. A 407 during the CONNECT tunnel is your proxy talking, not the website, so the fix is in your proxy config and never in the target's headers.

The schema is strict on purpose. Unknown fields are rejected, and responseSource should only say proxy or target when you actually know which layer answered, not because of the status code alone.

2. Never recommend an automatic retry

Every diagnosis comes back with likely causes, the next thing to check, and what it can't know. For the 407 above, the summary starts with this:

407 reports an intermediary authentication requirement. It does not identify a wrong password as the sole cause.

Retries got the strictest treatment. In the output schema, automaticRetryRecommended isn't a boolean. It's the literal false. The tool can only tell the agent what kind of retry thinking applies. Here's what comes back for a POST that timed out waiting for response headers:

"retry": {
  "automaticRetryRecommended": false,
  "category": "reconcile_before_repeating",
  "advice": "Do not automatically repeat POST/PATCH or an operation of unknown semantics. ..."
}

If a write timed out after it left your machine, the order might already exist. A missing response isn't proof that nothing happened, and the tool says so instead of letting the agent assume.

3. Credentials never go through the agent

Config templates use environment variable placeholders. No tool asks for a password as an argument, and no response ever contains one. If the agent never sees the secret, it can't paste it into a transcript, a log or a commit.

4. The one tool that touches the network is boring on purpose

Run locally, the server sends no telemetry and makes no network requests. There's one optional route check, and it stays off until you set IPVOLT_ENABLE_ROUTE_CHECK=1. Then it sends a single request through your proxy to one fixed endpoint, with a 10-second deadline, and that's all it can do. It can't be pointed at an arbitrary URL, so it can't be turned into a port scanner.

The hosted version keeps basic metrics: tool name, success or error, duration and toolkit version. Nothing about your requests.

5. Return structured data and text

Every tool has an output schema, and responses include both the structured result and the same result as text. Clients that read structured content get clean JSON they can pipe into the next step. Clients that don't still get a usable answer.

The trade-offs

Client versions are pinned. The diagnose tool only accepts the exact versions the rules were tested against: curl 8.22.0, Requests 2.34.2, HTTPX 0.28.1 and Playwright 1.63.0. That keeps the answers honest, but anyone on another version gets an error instead of a diagnosis.
The local package requires Node 24+. That rules out everyone still on Node 22.
The first npm release went out without build provenance. The next one should come from CI with provenance attached.

Try it

Use the hosted endpoint (https://mcp.ipvolt.com/mcp, Streamable HTTP, no signup or API key), or run it locally:

npx --yes @ipvolt/proxy-toolkit-mcp@0.1.0

Setup for different MCP clients is at ipvolt.com/mcp, and the code is on GitHub.

Full disclosure: I'm building ipvolt, a proxy service. The toolkit is provider-neutral and works fine without it.

If you've built MCP tools yourself: do you return structured data, text, or both? Curious what's worked for you.

DEV Community: Lucas