What 8,000 conversions a day taught me about proxy rotation

elvin 12 — Sun, 06 Sep 2026 19:11:58 +0000

I run a small audio conversion tool. Nothing clever — you paste a link, it pulls the audio, you download it. It handles somewhere around 8,000 jobs a day now, and about 91% of the people who start a conversion actually finish the download.

Getting there took five months and four bugs I would not have found in a test environment. This is a write-up of those four.

Stack, for context: Flask, Redis, RQ workers, yt-dlp, and a pool of rotating proxies.

The setup

Requests hit a queue. Workers pick them up, choose a proxy from a weighted pool, and stream the file to disk while the user's download is already in progress. If a proxy fails, the job retries on the next one.

That last part matters. The retry chain is what gets success rate from about 92% to about 99%. It is also where three of these four bugs were hiding.

Bug 1: two proxies that were secretly one proxy

I had two entries in the pool that looked like this:

gateway-a.provider.example:PORT_1
gateway-a.provider.example:PORT_2

Same host, different ports. Different exit IPs when I tested them individually, so I treated them as independent.

They were not. They share an upstream gateway. When one got rate-limited, the other was already in trouble — but my code kept routing traffic to it, because it was keying cooldowns and slot limits on the index in the pool array, and index 2 still looked healthy.

The fix was one line — key on the thing that actually enforces the limit:

python
def proxy_key(proxy):
# Same gateway, different port = same upstream limit.
# Keying on the pool index treats them as independent,
# so a cooldown on one never reaches the other.
return f"{proxy['host']}:{proxy['port']}"

Lesson: your identity function for a resource has to match whatever enforces the limit, not whatever is convenient in your data structure.

Bug 2: rotation running inside the request path

Rotating an exit IP on this kind of proxy means asking the upstream to cycle. That call takes 25 to 36 seconds.

I had it inside the job. So roughly once every N jobs, one unlucky user waited half a minute extra while a rotation completed.

It never showed up in the median. It sat in p95, and I spent a while blaming yt-dlp for it.

python

Rotation is maintenance, not part of serving a request.

Inside the job it blocked one unlucky user for 25-36s and

only ever showed up in p95, never the median.

threading.Thread(target=_rotate_if_due, args=(cfg,), daemon=True).start()

Lesson: if a maintenance operation is slow and periodic, it does not belong in the request path — even if it is "only sometimes."
**
Bug 3: the fallback that picked the worst option**

When every slot on a proxy was busy, my code fell back to the next one:

python
fallback = (proxy_index + 1) % len(PROXIES)

Looks harmless. It is not.

The pool still contained entries I had deliberately disabled by setting their weight to zero. Weight zero kept them out of normal selection — but modular arithmetic does not know about weights. Under load, the fallback path was routing traffic to a proxy that was failing essentially every request.

python

Weight-0 entries are disabled on purpose. Modular arithmetic

doesn't know that, so pick only from the trusted set.

fallback = next_trusted(proxy_index, TRUSTED_PROXY_INDICES)

Lesson: I had two mechanisms for "this resource is disabled" — a weight and a list — and only one of them was consulted on the error path. If you disable something, disable it everywhere.

Bug 4: retrying things that were never going to work

This one had the biggest payoff and the least cleverness.

My retry chain gave every failure three attempts across three different proxies. Reasonable for a timeout or a rate limit. Completely pointless for:

video deleted
video private
age-restricted
members-only

No proxy on earth fixes a deleted video. But I was spending three attempts and roughly eight seconds finding that out, on about 2% of all jobs.

python
PERMANENT = (
"Video unavailable", "Private video", "This video has been removed",
"age-restricted", "members-only", "Sign in to confirm",
)

def is_permanent_error(msg: str) -> bool:
# No proxy fixes a deleted video. Retrying it three times
# burns ~8s and occupies a slot on every other proxy too.
return any(p.lower() in msg.lower() for p in PERMANENT)

Two things improved at once. The user gets a clear answer in about three seconds instead of eight. And the rest of the pool stops being occupied by jobs that were always going to fail — so everyone else's p95 drops as well.

Lesson: a retry policy that does not distinguish permanent from transient failures is not a retry policy. It is a delay.

Where it ended up
Before After
Success rate ~92% 99.3%
Median ~11s 6.6s
p95 30-60s ~17s
Under 10s ~60% 89%

Not all of that is the four bugs — some of it is simply better proxies. But the p95 numbers are almost entirely bugs 2 and 4.

T*wo things I got wrong for a while*

Measuring per job instead of per attempt. I was computing proxy success rates from job outcomes. But one job can touch three proxies. A good proxy cleaning up after a bad one looked identical to a proxy that succeeded on the first try. Once I counted per attempt, one entry turned out to be sitting at 82% while I thought the pool was uniform.

Trusting one-hour windows. Success rate swings several points hour to hour. I made at least two changes based on a one-hour sample that a 24-hour sample would have told me not to make.

One last thing, if you are running yt-dlp in production: update it weekly. I have a cron job that pulls the latest build every Monday and restarts workers. Pinning felt safer right up until the first time an extractor broke mid-week and we sat down until someone noticed.