This is an open request for help/suggestions. I'm new to contributing to this community, but a long-time fan.
I have a problem that I'm hoping you can help with. I'd post this on Stack Overflow, but I'm hoping a conversation here can be more open-ended than it would be there.
I'm currently building a web app in which image uploading is a core component. I have uploading working and implementing, with a progress bar to show the user the status of the upload. I also have the ability to crop/resize the image after it's uploaded. I'm currently doing this all with the
multer npm package.
However, whenever I deploy my server, the PaaS I'm using wipes out all the uploaded files because the new deploy replaces the old one. So, I needed to switch to uploading to a 3rd party service like S3 (or start hosting my web app on Digital Ocean).
Recently, I got uploads to S3 working with
multer by using the companion package,
multer-s3. However, with
multer-s3, I lose the ability to track the progress of the upload. If you look up the
httpUploadProgress event that's fired by the AWS SDK, you'll quickly find that the event is only fired at the end of an upload -- not multiple times a second as you might expect from an AJAX
So, if I want to keep support file uploading and still have a progress bar, here are my options:
- Use a 3rd party service like Cloudinary or Transloadit or Uploadcare or Imgix, etc., etc.
- Upload images directly to S3 using a Presigned Url
Or.... 3. Get rid of the progress bar altogether.
For the life of me, I can't understand why this is such a hard problem. I feel like there must be a simple solution I'm just not aware of.
To quickly go through the reasons I don't like my current options:
A 3rd party service locks me into their API. Some of these services require jQuery and/or other large front-end libraries in order to upload directly to them (I'd prefer just an endpoint). And, if I choose to send the file directly from my server to avoid these extra client-side dependencies, it's not clear from their docs if most of these services would even return a progress event... However, in the end, the main problem with these services is cost. I don't ever want to pay $20 - $50 per month to handle file uploads if I can do it for $1-$2 on my own (although, at this point, it might be worth it).
The second option, uploading directly to S3 from the client using a Presigned URL, would be great in the near term, as I don't really want to modify the images just yet. However, in the near future, I could definitely see converting some of the images to WebP for browsers that support that image format -- or trimming some fat off the larger 2-20MB images.
And, as for the option of removing the progress bar... I think it's a huge usability enhancement to have one! As a user, it's really nice to know when your work is saved and how long it will take. I guess I could just show an infinite loading indicator, but then I worry about the people on a slow connection who might be sitting there with no indication when the stupid indicator is going to stop spinning... just waiting, and waiting...
So... Are there any services or packages you know of that I'm missing? Is there anything you know of that will let me upload files in Node.js to an 3rd party storage service like S3, while showing a progress bar to the user, while not adding lots of dependencies to my client-side JS, while giving me the ability to optimize/crop/convert images after they've been uploaded?
Now that I think about it... maybe this IS a lot to ask... heh. And, to think, I was incredulous towards a back-end developer 10 years ago when he told me this was a hard problem... 🙈
Any open source solution would be a huge plus, but an upload service that costs $10/month would be great too.
This is an open discussion and all suggestions are welcome! 😬😁☃️ Happy Holidays!