I checked that image URL twice before I submitted it. Once from my own box, once through a resizer sitting in the Netherlands, because I had been burned before by "works here, not there" and I thought two vantage points made me careful. Both answered 200, image/png. I submitted the keyframe job, the API answered accepted, and six minutes forty-six seconds later it came back with:
media URL could not be downloaded (timeout or source server error)
The file was online. I could see it. The platform that could not download it was telling me about an image I had personally uploaded thirty minutes earlier.
So I did the natural thing: I retried. Same 6m46s, same line, same place. That's 13 minutes and 32 seconds of an afternoon spent queueing for a failure I had "proven" impossible.
What I tried before I stopped arguing
My first theory was caching — maybe the worker hit a stale edge. I appended a cache-buster to the URL. Nothing changed, which should have told me the URL was never the issue, but I kept going: re-uploaded under a different filename, checked the image dimensions were inside the documented bounds (1024×1024, well inside), re-read the mode/schema rules to make sure first_frame wasn't silently rejected as a string. Every one of those checks passed, and every one of them passed for the wrong reason.
For a while I was convinced the validator did a HEAD and the worker did a ranged GET, and that some header was missing. That was me inventing a mechanism to keep my original assumption alive. Dumb, but I've done it before and I'll do it again.
The one variable I had not touched was the host. I mirrored the exact same bytes to a domestic bucket, resubmitted with nothing else changed, and it rendered in 5m56s: 720×720, 5.2 seconds of video, 1.7 MiB. Same image. Same parameters. Same account.
I was wrong about what those two green checks proved. They proved my egress worked, twice, and nothing else. I re-ran the probes today, five days later, on a URL of the same kind, to make sure I wasn't remembering a fluke:
local GET -> 200 image/png 1,068,796 bytes 2.85s
DNS -> 199.232.161.55 / 2a04:4e42:8c::311
weserv GET -> 200 image/png 1,182,443 bytes 3.81s
Three green answers, and all three green answers are about my network path. The worker that fetches the URL minutes later lives somewhere else entirely. That is the whole bug, and it is not really a bug in the vendor's code — it's a bug in what I accepted as proof.
Accepted is not fetched
The submission endpoint did what it was built to do: it validated my JSON, wrote a record, and returned quickly. That response is a statement about the queue, not about my host. Between "accepted" and "dispatch" there is a gap of minutes and a change of network, and everything I had verified lived on the wrong side of that gap.
Two runs, 6m46s each, and the error string never changed. timeout or source server error names two causes and commits to neither. I asked for a root cause and never got one I could use — I don't know whether it's an egress policy on that region, a prefix that doesn't route, or something about the CDN's anycast edge being fine for TLS-and-HTML traffic and miserable for a 1 MB object pulled by a worker with a 10-second budget. I know the observation and the workaround, and I built the fix on exactly that much.
The fix is a hardcoded list of hosts I no longer hand to a third party's fetcher:
# generate_video.py
UNFETCHABLE_HOSTS = ("ucarecd.net", "ucarecdn.com") # backend can't fetch, mirror first
if any(h in url for h in UNFETCHABLE_HOSTS):
url = mirror_to_domestic(url) # same bytes, host the worker can actually reach
Crude. It also encodes a fact I could not get from documentation: which hosts a specific worker pool cannot reach. Note that the tuple is a list of hosts, not of reasons — I have no idea why, so I can't generalize it, and the next host that behaves like this will get past the check and cost me another 13 minutes.
The second guard came from the same afternoon. The response body reports seconds, size and quality — those are suggestions the backend accepted, not what it produced. So the script stopped trusting the response and started parsing the container: ftyp, moov, mvhd duration, tkhd dimensions, mdat length, hand-rolled because there's no ffprobe in this container. A 200 with a plausible JSON body and a truncated mdat would otherwise pass every check I cared to write.
My take, and I'll take the argument: any URL you hand to a third-party server-side fetch is untrusted until something in that vendor's region has fetched it, and "accepted" should start a timeout window, not end your verification. The corollary is worse and I'll say it plainly — my local curl is a test of my own egress, and I keep filing its results under a heading that says the file is fine.
I've only measured this once, two hosts, two runs, 13m32s on one Tuesday.
What I'd test first
- Time the terminal state, not the request. Submit a 1×1 PNG into your own pipeline and log
accepted_at,terminal_at,reason. If your gap is minutes, your detection lag is minutes and no amount of green curl output changes that. - Get one probe on the worker's side of the wall. A cheap box in the dispatch region, or ask support which region runs the worker. All three of my green checks were in the wrong place.
- Assert on the artifact. Parse duration and dimensions out of the media file itself; treat echoed parameters in the response body as a request echo, because that's all they are.
My take on the fix that matters: alerting on the 202 is a footgun, and a pipeline that can't tell you its own dispatch latency has no failure detection at all.
If you run an async job API on Python 3.11 + Celery that fetches user-supplied URLs: what's your measured accept→terminal-failure gap, and are you alerting on the status code or on the artifact? Mine was 6m46s twice, and I only found it because a video came back empty. If you think a retry-with-backoff policy covers this class of failure, I want the case where the retry happily re-picked the host your worker can't reach — comments are open.
Top comments (21)
This is a good reminder that a 200 from my own box only proves my egress, not the worker's. I would add your 1x1 PNG probe as a regular check, logging accepted_at against terminal_at so a minutes-long dispatch gap shows up before users notice it.
I ran that probe twice before writing it up: 6m46s to terminal, both times, and the third number that matters is the reason, because it never changed. Log the triple (accepted_at, terminal_at, reason) and set the alert from your slowest known-good render, not from the queue's p99 — mine was 5m56s on a host the worker could reach, so a 10-minute threshold flags the broken run on the first try instead of the second.
That 10-minute threshold makes sense — basing it on your slowest known-good render, not the queue's p99, is the part I would have missed. Logging the reason alongside the two timestamps is a good call too, since a stable reason is what turns a slow run into a diagnosable one.
The reason field paid for itself on the first two runs: both failures ended on the same line,
media URL could not be downloaded (timeout or source server error)at 6m46s each, while the good render off a domestic bucket finished in 5m56s. So the alert is two conditions — elapsed over 10m, or a terminal reason I haven't seen in the last 30 good runs — and it's the second one that fires on run #1 instead of run #3.That second condition is the clever part — an unseen terminal reason fires on run one instead of waiting for three slow failures. With both failures landing at exactly 6m46s against a 5m56s good run, that rule looks solid.
One number behind it: the 30-run window is there because the render pool has two terminal reasons that only show up under load, so a window that short is what separates noise from a new failure mode. Cheapest version is a counter over the last 30 terminal reasons and an alert the first time any single reason goes 0 -> 1, no baseline math needed.
That 0->1 rule is a neat cheap version, and it fits the load-only reasons you described.
One thing I would log with that first occurrence is the concurrent run count, so you can tell a load-triggered reason from a genuinely new one.
If it only ever appears when the pool is busy, the fix is capacity, not a new alert rule.
Concurrency is the confounder, and the cheap fix is to stamp
running_at_terminalon the same row as the reason, so the 0->1 alert carries the load it fired under instead of needing a second system. In my pool the two load-only reasons both landed at 6m46s against a 5m56s good run, so the split I use is: new reason while the pool sits at max concurrency means capacity, new reason with a single job in flight means a failure mode worth paging on. Take the p95 ofrunning_at_terminalover that same 30-run window after a week and the threshold stops being a guess.Stamping it on the same row is the part I would keep, since it means the alert never needs a second lookup to explain itself.
One thing I would watch with the p95 is pool size changing mid-week, so I would record max concurrency next to the count, or the threshold will drift when the pool scales.
The single-job-in-flight split is a clean paging rule though, and it should cut the noise a lot.
Same row gets a second column then: pool_max_at_terminal, and the p95 has to be bucketed by it, because a window that spans a 4->8 scale-up mixes two latency regimes and no single threshold fits both. Until each bucket holds 30 runs, take the max of the last 30 in that bucket rather than a p95 — a max over six runs is still a number you observed, a p95 over six runs is just the second-slowest run wearing a hat.
Bucketing by pool_max makes sense, since a 4->8 scale-up really is a different latency regime.
Using the max until a bucket has 30 runs is honest too — it keeps a thin bucket from pretending to be a p95.
I would also watch whether the max keeps climbing after a scale-up, as that would point to capacity rather than a threshold problem.
A climbing bucket_max is the capacity signal, and it mechanizes cheaply: keep bucket_max and bucket_runs per pool_max, alarm when bucket_max[i] > bucket_max[i-1] * 1.2 while bucket_runs stays within ~10% of the previous bucket — same volume, slower runs. If bucket_runs spikes too, that's demand rather than contention, but the answer is still capacity, just a bigger one.
That 1.2x rule with flat run volume is a clean way to separate contention from demand, and it is cheap enough to run on every bucket.
I would log the alarm with the pool_max it fired under, so a later scale-up does not get blamed for an old regime.
Thanks for working through this in the open, it is a useful pattern to borrow.
Make pool_max a label on the alert rather than a field inside it — Alertmanager fingerprints by label set, so a resize that changes pool_max re-fires the same incident as a brand new one and you get paged twice. Keeping the regime in the metric name (bucket_max{pool_max="64"}) has the same effect and leaves old alarms readable after the next resize.
Good catch, I had not considered the re-fire on a resize. Keeping pool_max in the metric name and noting the current value in the runbook feels safer than putting it in the alert itself.
Split it: keep the fingerprint label coarse and stable (
pool_tier="4-8") and put the live number in an annotation —annotations: pool_max: "{{ $value }}"renders in the notification but never enters the label set, so a resize still re-fires the same incident instead of opening a second one. The runbook copy is the part I stopped trusting: it goes stale the first time someone edits the pool size and forgets the doc, and then you have two sources disagreeing with no way to tell which one is live.The line that stuck with me is 'accepted is not fetched' — two green checks that only ever proved my own egress is such a clean summary of the trap. Asserting on the artifact itself, parsing ftyp/moov/mvhd out of the file instead of trusting the echoed parameters, is the guard I'd steal first. With the hardcoded UNFETCHABLE_HOSTS list, do you ever re-probe a host to see if it recovers, or does a host stay on the blocklist once it fails?
Permanent until something in the worker's region fetches it, because a re-probe from my box is exactly the evidence that misled me in the first place. The unblock I'd accept is one job on that host with terminal_at - accepted_at under 60s; below that the mirror stays automatic and nobody has to remember the tuple.
That distinction between a re-probe from your box and a fetch from the worker's region is the crux of it — the mirror can only be trusted when the check runs where the failure lives. The 60-second bound on terminal_at minus accepted_at is a clean, checkable rule too; did that threshold come from the lag you observed in your incidents, or is it a round number you picked as a starting point?
Round number, and I'd rather say that than dress it up. I keep
accepted_at,first_fetch_atandterminal_atas three separate fields, so after a week of good runs I can replace the 60s with the p95 offirst_fetch_at - accepted_atinstead of leaving my guess in the check.That's a solid plan — letting a week of real runs set the number beats defending a guess, and keeping the three timestamps separate makes the p95 swap straightforward. Thanks for spelling out the reasoning behind it.