The first version of our nightly PDF batch job fired off requests as fast as the event loop could schedule them. It finished, eventually, but the logs were full of 429s, the total runtime was wildly inconsistent night to night, and once, memorably, it took almost three times as long as usual for reasons nobody could explain until we actually looked at the throughput graph.
Why unbounded concurrency looks fast and isn't
Firing every request as soon as it's ready feels like the correct way to maximize throughput, and for the first few seconds, it is. Then the rate limit kicks in, a chunk of requests come back as 429s, the client backs off, throughput craters, and the whole cycle repeats. The resulting throughput graph is a sawtooth: rapid climb, hard drop, rapid climb, hard drop, over and over for the length of the run. The average throughput across that pattern is meaningfully worse than a steady rate held just under the ceiling the whole time, because every throttled request is wasted work that has to be retried.
What bounded concurrency actually looks like in code
The fix isn't complicated, it's a semaphore limiting how many requests are in flight at once, tuned to sit comfortably under the rate limit rather than trying to hit it exactly:
import asyncio
SEMAPHORE = asyncio.Semaphore(25) # well under the 40 req/s ceiling
async def process_file(file):
async with SEMAPHORE:
result = await pdf_api.run_async({"action": "compress", "file": file})
return result
async def process_batch(files):
return await asyncio.gather(*[process_file(f) for f in files])
Twenty-five concurrent requests against a forty-per-second ceiling leaves headroom for natural variance in request duration, so the actual observed rate stays under the limit even when a few requests happen to take longer than average. Setting the semaphore right at the ceiling looks fine in theory and produces exactly the same sawtooth as no limit at all in practice, because request latency isn't perfectly uniform.
Backoff for the requests that still get throttled
Even with sensible concurrency limits, some 429s are close to unavoidable at scale, another process sharing the same rate limit, a brief spike in request latency that causes a temporary pileup. Retrying those immediately just recreates the problem. Exponential backoff with jitter, waiting progressively longer between retries, with some randomness so a batch of throttled requests doesn't all retry in lockstep, is what actually resolves this instead of just delaying it:
async def process_file_with_backoff(file, max_attempts=5):
for attempt in range(max_attempts):
async with SEMAPHORE:
result = await pdf_api.run_async({"action": "compress", "file": file})
if result.status != "rate_limited":
return result
delay = (2 ** attempt) + random.uniform(0, 1)
await asyncio.sleep(delay)
raise Exception(f"Gave up on {file} after {max_attempts} attempts")
The jitter matters more than it looks like it should. Without it, every throttled request from the same burst backs off on the exact same schedule, and they all retry at the exact same moment, recreating the spike that caused the throttling in the first place.
Sizing the batch job around the deadline, not the other way around
It's tempting to tune concurrency for the fastest possible completion time, and then find out the job occasionally fails outright when a burst of throttling pushes it past its window. A more useful framing, once there's an actual deadline involved, a batch that has to finish before business hours, say, is to size concurrency for a comfortable margin under the deadline rather than for maximum speed. A job that reliably finishes in forty minutes against a two-hour window is a better outcome than one that usually finishes in twenty-five minutes but occasionally runs long enough to miss the window entirely under adverse conditions.
Logging enough to actually diagnose a bad night
The incident that first got this project taken seriously, the run that took three times longer than usual, would have been diagnosed in minutes instead of a full afternoon of guessing if the concurrency and retry logic had been logging its own behavior from the start. Once the batch job started emitting a per-minute summary, requests sent, successes, throttles, current backoff state, a slow night became immediately explainable from the logs instead of requiring a live reproduction to understand.
Ramping up instead of starting at full concurrency
The controlled version in the throughput comparison doesn't jump straight to its target concurrency, it ramps up over the first several minutes of the run. Starting cold at full concurrency against an API that hasn't seen this client's traffic pattern yet is a good way to trigger the exact throttling behavior the concurrency limit was meant to avoid. A short ramp, starting at a low concurrency and increasing gradually while watching for early 429s, finds a sustainable rate empirically rather than guessing at one upfront.
What this actually bought us
The measurable outcome wasn't a dramatically faster batch job, it was a predictable one. Runtime variance across nights dropped from a wide, unpredictable range down to something consistent within a few minutes, night over night, because the job was no longer racing into the rate limit and getting punished for it on some nights more than others. Predictability turned out to matter more than raw speed here, since the batch job's only real requirement was finishing before a specific morning deadline, not finishing as fast as physically possible.
Where the API's own behavior matters
None of this concurrency and backoff logic requires knowing anything about how the PDF operations themselves work internally. It's entirely about being a well-behaved client of a high-concurrency PDF API that handles merge, split, compress, rotate, watermark, and convert as batch-friendly operations, priced per successful result, so failed or throttled attempts that eventually succeed on retry aren't charged twice. If your nightly batch job's runtime varies wildly for no apparent reason, check the throughput pattern before assuming the fix is more compute.
Top comments (0)