Scrapers love to live in containers. It makes deployment reproducible and scaling trivial. But the moment you move a working scraper into Docker, two proxy-related things tend to break: the proxy config that lived in your shell is gone, and DNS or networking behaves differently than on your laptop. Here is how to run proxied scrapers in Docker cleanly, so the container behaves the same in production as it did locally.
Pass proxy config as environment, not hardcode
Do not bake the proxy string into the image. It changes, it is a secret, and a rebuilt image should not require code edits. Pass it at runtime as an environment variable and read it in code.
FROM python:3.12-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD ["python", "scraper.py"]
docker run --rm \
-e PROXY_URL="http://user:pass@proxy.host:8080" \
my-scraper
Your code reads os.environ["PROXY_URL"] and passes it to the client. Now the same image runs against any proxy without a rebuild, and the credential stays out of your source.
Mind the difference between app proxies and Docker's proxy
There are two separate proxy concepts and they get confused. Docker itself can use a proxy to pull images, configured in the daemon. That is not the same as your scraper using a proxy for its requests. You almost always want the second, set in your application code or via the standard HTTP_PROXY and HTTPS_PROXY env vars that many HTTP libraries respect. If you set the daemon proxy expecting your requests to route through it, nothing happens to your scraper's traffic.
Standard proxy env vars
Many libraries automatically honor HTTP_PROXY, HTTPS_PROXY, and NO_PROXY. Setting them in the container is the least-code way to route traffic.
docker run --rm \
-e HTTP_PROXY="http://user:pass@proxy.host:8080" \
-e HTTPS_PROXY="http://user:pass@proxy.host:8080" \
-e NO_PROXY="localhost,127.0.0.1" \
my-scraper
Set NO_PROXY for local addresses so your container's own health checks and internal calls do not get routed out through the proxy, which is a subtle cause of hangs.
Scaling containers with a rotating pool
The clean pattern for scale is many identical containers all pointing at one rotating endpoint. Each request from each container draws a fresh exit, so you do not have to assign a different address per container by hand. If a container runs a stateful session, give that one a sticky exit instead. This keeps the orchestration simple: same image, same env, and the pool handles address distribution.
What the pool needs
WinGate suits containerized scraping well: private IPv4 proxies with SOCKS5 and standard auth that the env-var approach reads directly, a rotating pool that spreads load across a worldmix so scaled-out containers do not all share one address, unlimited traffic for long-running jobs, and support for up to 5000 threads. It speaks HTTP, HTTPS, and SOCKS5, so HTTP_PROXY and HTTPS_PROXY cover most libraries without extra code. There is a free 2 hour test, so run one container against it and confirm the exit rotates before you scale the replica count up.
An honest note: containerizing does not change how a site treats your traffic, and a proxy does not exempt you from its terms. Whether you run one process or fifty containers, pacing and rotation still do the anti-blocking work. What Docker gives you is reproducibility, so what you tested is what you ship.
The takeaway: pass the proxy as an env var not a hardcoded string, know the difference between the Docker daemon proxy and your app proxy, use HTTP_PROXY and NO_PROXY for the least-code route, and point scaled-out containers at one rotating pool. Do that and your scraper runs the same in a container as it did on your machine.
Top comments (0)