We added batch uploads to Longscribe (longscribe.com, free long-form transcription) so people could queue a folder of recordings instead of babysitting one file at a time. Simple feature on paper. It shipped a bug that took a full day to track down, and the root cause is a pattern worth watching for anywhere a self-healing job queue meets multi-file uploads.
The setup
Longscribe has a background "self-heal" sweep that looks for jobs stuck in a weird state (crashed worker, timed-out download, whatever) and retries them automatically. It's been solid for single-file jobs for months.
Batch upload changed the shape of the problem: instead of one job per submission, a batch is N jobs created together, meant to transition to "ready" as a group once all N files finish uploading.
What went wrong
The self-heal sweep runs on a timer, independent of the upload flow. It was gating on "does this job look stuck," which for a freshly-created batch job looks identical to a job that's stuck waiting on a slow upload.
Here's the sequence that broke things:
- User selects 5 files, batch upload starts
- Files 1-3 finish fast, files 4-5 are still uploading (bigger files, slower connection)
- Self-heal sweep runs mid-upload, sees jobs 1-3 sitting there "not yet dispatched to the transcription worker" and dispatches them
- The batch-upload completion handler also dispatches jobs 1-3 once the whole batch reports ready
Same three jobs, dispatched twice, by two code paths that had no idea about each other. Double transcription minutes burned, duplicate rows in the jobs table, and a confusing support case where a user swore they only uploaded once.
The fix
The honest fix wasn't "add a lock" - that just hides the race under load. It was giving the self-heal sweep a signal it had been missing: don't touch a job that belongs to a batch until upload_status is "ready" for the whole batch, full stop. A job mid-upload isn't "stuck," it's "not started yet," and those need different handling.
def is_eligible_for_self_heal(job):
if job.batch_id and job.batch.upload_status != "ready":
return False
return job.status == "stuck"
One line of actual logic, guarded by a state the self-heal code wasn't previously aware existed.
The lesson
Any time you bolt a new multi-step flow (batch, multi-part upload, saga, whatever) onto a system that already has an automated retry/self-heal/reconciliation process running independently, go find that process and ask it explicitly: what does your state machine think a mid-flight item in my new flow looks like? It probably looks like something you already have a name for - "stuck," "abandoned," "failed" - and your automation will cheerfully act on that wrong guess unless you teach it the difference.
Cost a day to catch in testing. Would've cost users double the wait and a support ticket instead, if it had shipped.
Top comments (0)