Hello Dev.to,
This is an open request for help/suggestions. I'm new to contributing to this community, but a long-time fan.
I have a problem that I'm hoping you can help with. I'd post this on Stack Overflow, but I'm hoping a conversation here can be more open-ended than it would be there.
I'm currently building a web app in which image uploading is a core component. I have uploading working and implementing, with a progress bar to show the user the status of the upload. I also have the ability to crop/resize the image after it's uploaded. I'm currently doing this all with the multer npm package.
However, whenever I deploy my server, the PaaS I'm using wipes out all the uploaded files because the new deploy replaces the old one. So, I needed to switch to uploading to a 3rd party service like S3 (or start hosting my web app on Digital Ocean).
Recently, I got uploads to S3 working with multer by using the companion package, multer-s3. However, with multer-s3, I lose the ability to track the progress of the upload. If you look up the httpUploadProgress event that's fired by the AWS SDK, you'll quickly find that the event is only fired at the end of an upload -- not multiple times a second as you might expect from an AJAX progress event.
So, if I want to keep support file uploading and still have a progress bar, here are my options:
- Use a 3rd party service like Cloudinary or Transloadit or Uploadcare or Imgix, etc., etc.
- Upload images directly to S3 using a Presigned Url
Or.... 3. Get rid of the progress bar altogether.
For the life of me, I can't understand why this is such a hard problem. I feel like there must be a simple solution I'm just not aware of.
To quickly go through the reasons I don't like my current options:
A 3rd party service locks me into their API. Some of these services require jQuery and/or other large front-end libraries in order to upload directly to them (I'd prefer just an endpoint). And, if I choose to send the file directly from my server to avoid these extra client-side dependencies, it's not clear from their docs if most of these services would even return a progress event... However, in the end, the main problem with these services is cost. I don't ever want to pay $20 - $50 per month to handle file uploads if I can do it for $1-$2 on my own (although, at this point, it might be worth it).
The second option, uploading directly to S3 from the client using a Presigned URL, would be great in the near term, as I don't really want to modify the images just yet. However, in the near future, I could definitely see converting some of the images to WebP for browsers that support that image format -- or trimming some fat off the larger 2-20MB images.
And, as for the option of removing the progress bar... I think it's a huge usability enhancement to have one! As a user, it's really nice to know when your work is saved and how long it will take. I guess I could just show an infinite loading indicator, but then I worry about the people on a slow connection who might be sitting there with no indication when the stupid indicator is going to stop spinning... just waiting, and waiting...
So, I'm really stuck here. On the one hand, I feel like image uploading with a progress bar (and the ability to modify the image on the back-end after) should be a solved and simple problem. While, on the other hand, I'm strongly considering just biting the bullet and going with something like Cloudinary, even though it will probably double the size of my client-side JavaScript bundle (something I'd really love to avoid).
So... Are there any services or packages you know of that I'm missing? Is there anything you know of that will let me upload files in Node.js to an 3rd party storage service like S3, while showing a progress bar to the user, while not adding lots of dependencies to my client-side JS, while giving me the ability to optimize/crop/convert images after they've been uploaded?
Now that I think about it... maybe this IS a lot to ask... heh. And, to think, I was incredulous towards a back-end developer 10 years ago when he told me this was a hard problem... 🙈
Any open source solution would be a huge plus, but an upload service that costs $10/month would be great too.
This is an open discussion and all suggestions are welcome! 😬😁☃️ Happy Holidays!
Latest comments (5)
Hello David,
Just by googling, I wonder I came back to your post. Ok I was having the same problem but I did figure out a couple things, that might be helpful to you.
My Use Case:
Angular ---> NodeJS ---> S3 Bucket AWS
S3 ---> NodeJS ---> Angular
From Angular to NodeJS it is simply a HTTP Event Progress
From NodeJS to S3 there is httpUploadProgress from S3
So to relay progress from NodeJS of S3 to Angular, I am using Socket.IO by creating an event and emitting it, this solves the problem of upload progress being emitted.
0%-----Angular to NodeJS----50% ---- NodeJS to S3----100%
If you need any further help, let me know, I am gladly to let you in for the solution.
Hi David. Hope you're enjoying the holidays.
Interesting problem you have here. So, I'd suggest using Cloudinary to handle media uploads and transformations in your application. This is mostly because I've had some experience using their service and I believe it will give you a hassle-free way of uploading images using just an upload endpoint thus, not increasing the bundle size of your application in any way. Let me highlight what advantage using Cloudinary gives you:
Another thing to note is that with Cloudinary Upload Presets, you can tie different transformations to an upload preset. For example, if you wanted to crop every image, you no longer have to create another function to do that every time. You can just tie the transformation to the upload preset and once you make an upload, Cloudinary handles the media transformation for you.
With regards to the cost involved, with Cloudinary, you pay based on your usage. This article explains in detail how the transformation quotas work.
Hope this helps!
Oreoluwa! This is the perfect solution!!
I've worked with Cloudinary before, so their API is pretty familiar to me. However, I had no idea about their support for uploading directly to an endpoint (I should've known!).
Thank you for this!
After a lot of research, this is the solution I'm going to use. It's just soooo convenient and sooo easy compared to all the other solutions I've researched. And, having worked with Cloudinary in the past, I'm very impressed with the power of their API (like being able to generate zip files from multiple files on the fly -- so users can download them all at once.
Thank you very much!
I still wish there was an easy way to do this on my own, but this solution will definitely suffice for the foreseeable future.
Happy Holidays David! In terms of premade software there is a windows tool you may be able to use called S3 Browser (windows only I believe). I haven't used it directly but AWS cert instructors have suggested it to me in the past available here.
I can throw in a couple points here that might be worth considering as well:
Giving the ability to optimize/crop/convert images after they've been uploaded is a particularly difficult and potentially expensive feature to try include. Once something is uploaded to S3 you will be charged for downloads/modifications of the file in accordance with their pricing scheme. Maybe consider having users pre-process their images before going through the upload cycle. Otherwise you'll have users uploading an image, redownloading it, editing it, reuploading it and need logic to delete the initial image. Seems like too much power for the average end user!
I understand the reluctance to rely on third party libraries but I don't think a web app is the optimal way to handle this -> especially if your images are large files and you want users to upload them in batch. You might consider building a separate microservice to handle the file uploading that your main web app calls. Keep the logic separate in case of issues. Also maybe a CLI tool is a better alternative with prompts etc. I've only used commander in Python but I know there are JS options available (maybe even make it an electron app users can download locally!) Have it call out to the AWS API.
If your users are uploading multiple files you can always build a makeshift library to aggregate the progress based on whatever data you can get out of the AWS API. I'm sure it gives file success uploads after each files uploaded, mix that with some logic deducing the remaining filesize left to go to setup a rough proress bar? There must be a few progress bar libraries on NPM you can make use of for that as well to reduce the workload.
Hope the above is of some help or at least useful food for though.
Happy Holidays!
Hi Daniel! I really appreciate this response!
I've done some searching based on your serverless suggestion and found some amazing options.
One option is a Serverless Image Handler directly from AWS. I think it would let me upload an image directly to an S3 bucket (thereby getting a progress bar) and then serve these images with Cloudfront.
However, the huge bonus is that I think it detects whenever an image is uploaded and uses the API Gateway service (through a Lambda function) to modify the image according to one of these transformations.
The second option I found through a dev.to post, which walks you through creating a serverless web app service that can upload images to S3 and then modify them. I'm not positive if I'd be able to get the progress bar with this solution, but I think I probably could.
However, after all this research, I decided to go with Oreoluwa's solution, which he posted in a comment below. I decided that I'd rather not build/maintain a serverless app in addition to my own app -- and with my usage, Cloudinary will be free for me for a while. I could see switching over to one of the options above in the future when I get a bigger tech team, but right now it's just me and building/marketing/selling one service is already too much. I don't want to take on a whole new stack (i.e. serverless) that I don't have a lot of familiarity with yet.
Anyways -- thank you for your help! It's much appreciated!!