(Feel free to connect on LinkedIn Uzair Aslam and by emailing uzairaslam043@gmail.com/uzairaslam196@gmail.com)
1. Introduction
In the realm of web development, certain challenges often lead to ingenious solutions. One such journey unfolded when tasked with implementing a feature for users to upload image files in live views. This blog narrates the twists and turns faced in achieving resumable uploads, concurrent gallery uploads, and uninterrupted user experiences.
2. The Challenge: Resumable Uploads and User Flexibility
The primary goal was clear—users should upload batches of images (500+) seamlessly. If the upload process got interrupted, say due to connectivity issues, due to system restart, or due to server crash, or server redeployment, it needed to pick up right where it left off once the connection was restored. Furthermore, users should have the freedom to navigate to different LiveViews without disrupting ongoing uploads. Notably, users have the option to refresh the view, but the uploading process should intelligently resume from the same point, ensuring a seamless and uninterrupted user experience. I did some R&D but I was unable to find this kind of implementation on any website. Even I explored YouTube's feature of video uploading but they also don't support resumable uploading.
3. The Shift from allow_upload to Manual Implementation
Initially, we were using the allow_upload feature provided by LiveView, which was fantastic. However, as our requirements evolved, this method no longer fit the bill. The shift to a manual approach using socket events to send files information to the server to build their presigned_urls and then send them to the client to start uploading. Similarly, socket events to keep track of progress and when any chunk of file is uploaded.
4. Optimizing Uploading Initial Wait: A Chunky Strategy and Elixir's Task Module
Imagine a user selects 1000 files and sends information to the server. The task of sending pre-signed URLs at once for each file could prove time-consuming. To tackle this, we came up with a smart chunking strategy. By dispatching pre-signed URLs to the front end in batches of 100, we significantly hastened the display of files in the DOM, eliminating the need to wait for all URLs simultaneously.
The process of generating pre-signed URLs on the Elixir side in a loop for 1000 files might have been sluggish. Enter Elixir's Task module, equipped with a max_concurrency setting of 100. This ingenious addition allowed us to concurrently process 100 files at a time, swiftly fetching and sending pre-signed URLs to the client.
5. Overcoming Storage Challenges: The Shift to IndexDB
The initial strategy was centered around tracking file paths, data, and checksums using Chrome storage. Unfortunately, we hit a roadblock due to permission issues. Our original intention was to fetch files from the file system using saved paths for resumption upon internet restoration. However, it became apparent that the file system doesn't permit the retrieval of files from its own repository, necessitating the less-than-ideal action of users reopening the popup window to select files.
In response to this user-friendliness concern, we strategically transitioned towards utilizing IndexDB. This shift empowered us to store entire files within IndexDB, providing a robust and seamless approach for resuming uploads without the need for users to re-open the popup window. This adjustment not only addressed the initial hurdles but also ensured a more user-friendly and efficient experience for file uploads.
6. Enter gcs-browser-upload: A Resumable Upload Savior
The discovery of the gcs-browser-upload JavaScript library was a turning point. This library streamlined the resumable upload process, requiring only the same URL to seamlessly resume a file from where it left off. It handled checksum tracking, simplifying our workflow.
7. Background Uploads: Sticky LiveViews
Live View provides an option to stick a live view across different live views. We can provide an option sticky: true in render(ChildLiveView, options). The sticky LiveView ensures uploads continue in the background, providing a smooth user experience.
8. Tracking Progress of All Galleries: GenServer
The server receives information from the client on each chunk upload of a file. It stores the progress of each file in assigns and keeps track of the overall progress of all galleries, a separate GenServer process was implemented. This allowed tracking the progress of each gallery independently and showcasing an overall progress indicator on the page.
9. The Seamless Flow: Resuming Uploads
Picture this scenario: a user selects files, makes progress, and decides to refresh the page. Thanks to the dynamic duo of IndexDB and the gcs-browser-upload library, upon reloading the page or restoring the internet connection, all pending files are seamlessly retrieved from IndexDB. The magic happens when the same pre-signed URLs are provided, in this way, the upload process resumes seamlessly from the exact point where it left off.
10. Gratitude for Collaborative Effort
A special credit goes to a colleague who initially worked on the JS side, contributing valuable insights and ideas that paved the way for this resilient and user-friendly solution. In actuality, he provided me with the initial implementation of JS to store files and resume them from the same point. Here is his LinkedIn profile: Omar.
Conclusion: A Symphony of Ingenuity and Collaboration
In conquering the complexities of file uploads, the integration of technologies like IndexDB, Elixir's Task module, and the gcs-browser-upload library resulted in a symphony of ingenuity. The seamless flow and resilience in the face of challenges exemplify the power of simplicity and adaptability in crafting robust web solutions. I will create a separate Repository with a minimum implementation of it and will add the link to that repo here in the coming days.
Top comments (0)