The Beginning
My sister asked me to create an application for her wedding where guests could upload images and videos. At the time, I had only been working on backend development for about a year or two. My experience was limited to building simple APIs, collecting form data, and storing it in MongoDB with a Node.js backend.
I decided to take on the challenge. What started as a small project eventually grew into a full application hosted on a VPS, running in a Docker container, using S3 for media storage and PostgreSQL to track uploads. This is the story of how I got there.
Planning
The first version of the app was just a basic form that uploaded images to Cloudinary. At this stage, I had to learn what a file actually was in programming terms. I only vaguely understood blobs, and realising that a file was a subclass of a blob, this took me some time to wrap my head around.
Initially, I thought I could just send an image directly to a Node server and then forward it to an API. That’s when I discovered Multer and its quirks. With it, I managed to get a simple upload working with Cloudinary.
But Cloudinary’s free plan had strict limitations, especially for video uploads. Since I wanted guests to upload both photos and videos, I started researching alternatives. At this time I never touched any cloud storage after hearing the horror stories I heard of people making mistakes and being charged a lot of money for Lambda functions I didn't want to try S3 bucket.
I briefly considered Google Drive. My thought was, if I can store videos there, why not use it for my app? But after digging into the documentation, I realised how locked down Google’s API was made it very overwhelming at the time. On top of that, Google Drive has a soft limit of three write requests per second. That meant if multiple guests uploaded at once, errors could occur. This wasn’t acceptable for the application, so I had to move on.
S3 Bucket
As I started to run out of ideas for a place to store the media files, I looked towards AWS and tried to get around the fear of causing a massive mistake and tried it out. So I began my research on AWS watched guides, tested out creating an account and creating small projects to see how it worked. This is where I learnt how to chunk a video into blobs send that to my backend to then convert to a buffer that streamed to S3 Bucket. This went on for a few months as there were a lot of concepts I wanted to understand and some secure ways to upload files to AWS.
At this point I was confident to use AWS as my main storage and to remove Cloudinary from my project. At this point I had the basic functionality working where quests can upload images and videos using the @aws-sdk/client-s3 single file for image uploads and multipart upload for videos.
UI Design
Up until then, the UI was just a plain form. I wanted something simple and intuitive so even non-technical guests could use it without issues. After researching different designs, I came up with this layout:
With this design I wanted to give guests as much information as possible without overwhelming them. How it works is when a guest wants to upload files they can select the button at the top that says Add files. When they have selected their files each file will be listed out with the name, size and type.
This is a preview of the files the guest has selected to upload. Each card is also provided with an X button that allows guests to remove files from their upload. The file limit of 20 files and 5GB of space was added later for handling bad actors, We will discuss this later on.
With the file upload when you try to upload more files it usually replaces the files on the form, so I had to do some comparing and filtering to prevent duplications and removing of the wrong files. Under the list of file there is a drag and drop box that allows guests to drag and drop files into the list for guests on PC.
As I was designing each part more ideas and problems kept rearing from each part of the application,
- How do I handle errors?
- How do I let the guest know the file is still uploading?
- How do I handle bad actors? Do I get guests to log in?
I decided to use toasts for sharing information with guests, as I said before I wanted to make this as user-friendly as possible. Toast will pop up at the top of the screen to inform guests of errors or successful uploads.
Handling the UI for the uploading was a bit tricky as I tried a few different styles as when your uploading single files updating the UI and informing guests was fine, but when dealing with videos and chunking it would be stuck on the same file for a while so I had to plan how I display to the guest its still uploading the video file is just taking a bit longer.
I planned on uploading the chunks to the server one by one to help show guests that the file upload is still in progress. I used the file's memory sizes to compare them to the uploaded ones and created a progress bar that moved with the memory being updated as the video chunks are being uploaded.
The part that caused a lot of head scratching was how to deal with bad actors. I didn't want the guests to log in, but I also wanted to set a password to prevent unwanted uploads. So I decided that we would have a password that would be available on the day and I also had a QR code that was a URL with the password on it. When people scanned the QR code it would fill the password input box with the password so the guest only had to worry about the files.
I also added the file limits to prevent large uploads from bad actors who somehow gained access to the password. But if they didn't obtain the password they could spam the password input, so I placed a limit on wrong input with longer and longer bans for each mistake. This was more of a hurdle than for security as I wanted to make the upload as simple as possible for guests.
Hosting
A couple more months have gone by the application is working and I planned on testing it online. I decided to host it on something like Render or Netlify as I had projects hosted there. It worked fine but I felt I didn't have control of the backend and I wanted to make sure the servers were close to the event as uploading media added time to upload already and having a server hosted in America would delay uploads more.
I looked at other options. First I looked at AWS but the more I researched it seemed overkill for my application, so I started looking at VPS as I was already using the cool project Coolify to host some of my projects but I had a very limited idea of how they worked using guides.
I set up a new server and hosted my application and did some tests, it was working and I was satisfied, but the more I tested it the more I wasn't satisfied with using console.log as my main source to see errors and functionality of my application.
I wrote some code to stream my errors and successes to files to help keep track of what was happening. (Yes, I know I could have used Morgon or Winston, but I love to learn how things work and wanted to try creating my own logging) But at the time I noticed I wasn't able to access the files that were being generated on the Coolifty server. From my guess they were using Docker and I had no clue about Docker, so I wasn't satisfied that I couldn't read my logging files and decided to try managing a VPS.
VPS
This is not a sponsor but this course on Frontend Masters from Jem Young Full Stack for Front-End Engineers, v3 and Dreams of Code video Setting up a production ready VPS is a lot easier than I thought. to be a massive help in understanding on how to set up a VPS.
I got my VPS up and running with its domain name and reverse proxy. I had plans to have multiple servers with a load balancer or multiple Docker containers, but I decided on just one container for the application. At this point I also wanted to store data in a database to help track the amount of data being uploaded to help safeguard the S3 bucket from bad actors uploading an excessive amount of data. This was done by checking the amount of memory uploaded and if it's over it prevents any further uploads.
I was going to use MongoDB as I was comfortable with it but I wanted to use my SQL knowledge and decided to use PostgreSQL as my database. This meant I needed to use Docker Compose to allow my database and application containers to communicate.
Once I was happy with the application, I did some testing to see if I could get the application to break. I got family and friends to upload at the same time with different file types and devices and noted down any bugs or UI mistakes. I fixed the problems and rescaled the server as with our tests we got close enough to the max CPU. So to prevent any crashes I rescaled my server.
Outcome
The Wedding day came around and it was a lovely day. The application worked flawlessly with a few people typing in the wrong password, but apart from that it was up all the time and people understood how to use it without me hand-holding them. We received over 150 photos and 11 videos.
What would I change
- I would rewrite the backend handling uploads it got a bit messy later on and caused some bugs that took some time to fix.
- Write tests, I had some bugs that were so deep in information I couldn't see where it was breaking even when I was using debuggers it was tripping myself up.
Conclusion
I learnt a lot about how to handle media uploads and the pitfalls to look out for next time. I am more confident to test out things, even if it's an afternoon project I throw away after you might learn a thing or two. There was probably some things I glazed over that probably deserves more detail but the code is down below on my GitHub.
Top comments (0)