TL;DR: Too many downloads were being made from my database (due to ignorance and many bad decisions) which almost maxed out my free monthly limit of 10gb in only 2 days.
Hello world! This is my first post so I'm a bit nervous, but here's a funny story I thought I'd start things off with. I had just delved into using Firebase, so I was very ignorant on a lot of things.
Recently I've been working on a blog app. In the past I had made a very minimal blog app following a Udemy course, but this time I wanted to start from scratch and use some new tools I had acquired since then. I had just finished implementing a feature that allows each user to upload their own profile picture that is displayed throughout the site.
Since I was already using Firebase's Realtime Database to store some data, I decided to just store the images in there as well temporarily. This was my first mistake. My second mistake was sharing a link to the app with my family, thinking it would help me see if the site was user-friendly. Be careful what you wish for 🤷🏻
A little background on how my app is set up.
- A homepage that displays all posts.
- Each post consists of a title and body, and shows the author's avatar.
- A page that allows you to create a new post.
- A profile page that displays a user's posts along with a bigger image of their avatar. When logged in, your profile has a link that lets you edit your username and avatar.
- A navigation bar that displays the current users avatar.
So I received a warning email from Firebase on Sept. 10th telling me that my Realtime Database was at 8.8gb, and that I only had 1.2gb worth of downloads left for the rest of the month.
Here's the two main factors that caused this:
- I was storing the user avatars in base64, which lead to bigger file size. This meant downloading it from the database took longer and ate up more of my allotted downloads.
- Each time an avatar was being displayed, I was making a new fetch to the database, and not storing it anywhere. So each time a user visits the home page, new call is made to the database to download all the avatars that are displayed on each post.
Lets say each avatar is originally 170kb, which is uploaded to the database as base64, so the image size is now 225kb. That doesn't seem too bad. But if there's 30 posts on the homepage, 30 avatars are being downloaded from the database, so 30 * 225 = 6750kb. Still doesn't seem too bad, but lemme give you an example to find out how I tortured the Realtime Database.
Let's follow a typical user as they sign up and navigate my app.
A user visits my site for the first time. They are sent to the homepage, where there are 30 posts and 30 avatars are downloaded to be displayed on each post. That's 6750kb worth of downloads. This user decides to create an account, which redirects them back to the home page afterwards. 6750 + 6750 = 13,500kb. They go to their profile page to change their profile pic, then decide to check out the homepage feed again, and all 30 avatars are downloaded once again. 13,500 + 6750 = 20,250kb. They decide to write a post, and are redirected to the homepage afterwards. Again, +6750. Our total is now 27,000kb, which is 0.027gb. This still doesn't seem too bad, but don't forget that this is just one user.
If there are 20 people signed up on my app, and they all are navigating my app like in my example, that's 20 * 27,000kb = 540,000kb. Not to mention that's only one session! So if all 20 people visit my app 4 times per day (that's wishful thinking 🥺), that's 2,160,000kb, or 2.16gb in one day!! Don't forget that this whole example is assuming the images are all 225kb each. 😱 Thankfully my dear family decided to make multiple accounts and upload huge HQ images haha.
But most importantly, since I'm working on this app all day everyday, I'M the one using it the most. 😭 At the moment I don't have separate databases for development/production. In addition I'm using webpack-dev-server, so the app was refreshing each time I saved a file. All the while making real calls to the database to make downloads.
Anyway, here's how I solved this problem 😊
- Most importantly I'm now using Firebase's Storage for images, and the Realtime Database is solely for storing usernames and post info.
- I'm no longer storing the image in base64, instead using the original files format.
- Once an avatar is uploaded, I'm storing it's Firebase Storage URL in Redux, so the image is downloaded from the Redux store the next time it needs to be displayed.
This was already enough to decrease the downloads, but there's a few more features I'm working on that will greatly help improve things.
- Allow users to crop their image, and then a step further would be somehow compressing the image. Only then will it upload to Storage.
- Persisting the Redux state to local storage. If you visit the homepage, all the avatar URLs are downloaded and saved to the Redux store. This only works during one session. If you refresh the page or visit my app again another time, the store is reset, and all the URLs need to be downloaded from the database again. Saving the state in the browser's local storage fixes this. The next time you visit my blog app, the browser remembers what data has already been downloaded.
So anyway, that concludes my first big production bug. Rushing to do damage control so my database wouldn't be shut down for the rest of the month. 😅 Thankfully, ever since I switched to Firebase Storage and storing the image URLs in Redux/local storage, my Realtime Database still shows that I have 1.2gb left. Hopefully this means that my app will still be up and running for the rest of September!
Oldest comments (10)
Potentially YUUUUUUGE savings on image transfer and load times:
anothermadworld.com/how-to-put-a-c...
That's an interesting work-around! Might need to implement that if more people decide to use my app haha 😊 Thanks!
It's worth implementing anyway; not only for the speed advantage, but so you don't run up against Firebase Storage's free tier transfer limits.
Cropping and resizing image to a set standard will definitely help with storage limits!
agree! currently working on this 😊
Awesome first write up!
thanks! 😃
Am also building my blog from scratch
Not using firebase though
Was thinking of using base64 for my post thumbnail and users avatar 😂😂😂
But this kinda an eye opener
Funny question
how fast was your page loading before you fixed it ?
lmao 😂 before i fixed it the page load took several seconds. now it's noticeably faster, probably because i'm only downloading a small URL as opposed to a giant base64 string.
good luck with your blog!
Another thing to consider: Realtime Database is the old Firebase DBaaS solution. Firestore is newer, recommended going forward, and you can pull down individual collections or even single documents instead of syncing the entire database.
Still keep your images out of the DB though.