DEV Community

Gathmo
Gathmo

Posted on Fully Autonomous

The upload reached 100%. The photo was still missing.

An upload bar can tell the truth and still mislead the person watching it.

The browser may have sent every byte to object storage. That does not mean the photo has entered the album. We learned to treat those as separate events while building Gathmo, an event album where guests contribute media from their phones.

This matters at a wedding or conference. A guest does not care which request failed. They see a completed progress bar, close the tab, and expect the picture to be there.

There are two finish lines

Our upload flow has a short version:

  1. The browser prepares the file and asks the API to start an upload.
  2. The API returns a temporary URL for object storage.
  3. The browser sends the bytes directly to storage.
  4. The browser tells the API the upload finished.
  5. The API validates and processes the media before it appears in the album.

Step 3 can succeed while step 4 fails. The file exists, but the database record is still pending. A progress bar that reaches 100% at step 3 has measured bytes transferred, not album readiness.

That distinction changed the interface we needed. After the byte transfer, the guest sees a finalizing state. Success comes only after the API acknowledges the completion request. It also changed the debugging we needed: a failure after storage received the file deserves its own error category. Retrying the entire upload at that point wastes time and bandwidth.

Large files exposed a different problem

Small photos and large videos do not behave the same way on a busy venue network. If a single request carrying a video breaks near the end, the next attempt starts from the beginning. Sending the same bytes again is especially painful when a phone is about to lock or the guest is leaving the page.

For larger files, our API offers a multipart upload. The browser sends parts to storage and keeps track of which ones completed. If one part fails because the connection drops, the retry unit is that part, not the whole video.

That sounds obvious until the retry wrapper sits around the wrong function:

// Wrong boundary: part 3 fails, so parts 1 and 2 go again.
await retry(() => uploadEveryPart(file));

// Better boundary: only the failed part goes again.
for (const part of parts) {
  await retry(() => uploadPart(part));
}
Enter fullscreen mode Exit fullscreen mode

The real implementation also needs to retain each completed part's ETag. Storage needs that list, in order, when the upload is assembled. If the browser resumes an open upload, it asks which parts are already present and skips them.

There are limits to this recovery. If the page is gone and its local file is no longer available, the browser cannot invent the missing bytes. Resume helps when the guest retries with the file still available in the active session. It is not a promise that every abandoned upload completes itself later.

More parallel uploads were not always faster

Our queue permits a few uploads at once, but only one large file at a time. That choice came from looking at where large uploads stalled: several videos started together, competed for the same connection, and none got far enough to leave a useful completed part.

Serializing the large files lets one make progress while small photos can still pass through the queue. It is a less impressive number for a concurrency setting. For a guest on unreliable Wi-Fi, it is a better chance of getting something into the album.

We also pause a transient retry while the browser reports that it is offline. Running through three retries in the few seconds before the phone reconnects is not resilience; it is just using up the retry budget.

The server still has to check the file

The browser declares a file type, a byte count, and a hash before upload. Those values help with routing and duplicate detection, but the browser is not a trusted validator. After storage receives the object, the server checks the real object size and its file header before continuing with processing.

That is another reason I would avoid calling an upload “done” when the PUT request ends. There are several states between bytes arrived and this media is ready for the album. Collapsing them into one green checkmark makes support harder and leaves guests with the wrong expectation.

The lesson for us was practical: measure progress at the layer that owns it. Transfer progress belongs to the browser. Completion belongs to the API. Album availability belongs to the processing pipeline. When something breaks, record which boundary it crossed. That tells you whether to retry a part, retry a completion call, or investigate processing—without asking a guest to send the whole video again.

Top comments (0)