Anatomy of a fast Next.js job board

Angel Nikolov — Mon, 06 Dec 2021 14:42:28 +0000

Anatomy of a fast Next.js job board

Hello everyone! I’ve been working remotely for quite some time now and I also have a couple of failed side-projects behind me. Recently, I realised that I should probably invest time in new projects only if they made practical sense for me.
Then it hit me that the very thing that made it possible for me to work on side-projects was actually the thing which made sense for me to optimise and put more effort and ideas in. And that was remote work. One issue with remote work kept coming at me and that was — finding my new work place. I’ve used many different job boards, but all of them had different filters, job selection, mailing lists and etc. which obviously wasn’t ideal.

That’s when I figured that maybe it will be nice to have all of the most used job boards aggregated and presented to users in a nice, fast and efficient way. I purchased several domains which I used to further refine the choice of the user by job categories.
https://www.remotefrontendjobs.com and https://www.remotebackendjobs.com are now listing thousands of jobs, aggregated from more than 14 different sources (all linking back). People could also subscribe to weekly tailored notifications for new jobs (they can also specify if they are only interested in jobs with salaries specified).

In this blog post I will go through how I created a fast, beautiful and tailored job-hunting experience running on Next.js and Vercel ❤️.

To keep you interested, these are the tools that I’ve used for the first version of my job board which I will go through in this article.

When I started out, I wanted to pick the set of tools which would enable me to create a fully functioning job board in a quick and predictable manner. Initially, this was a really quick and neat idea in my head which would’ve taken me no more than a couple of hours. Basically, what I had in mind was a simple page with a list of jobs and a search bar on it.

So what I started with was a simple SSG (Static site generation getStaticProps) page in Next.js and a couple of Serverless functions which I can use to get the actual jobs. Whenever a user would navigate to my page, they would load the list below, which would be generated at build time by scraping some job boards.

There are a couple of important things to note in the code above.

I am using GetStaticProps to render this page at build time. This basically renders the whole application on the server and outputs a plain old and heavily optimized HTML pages which should load as fast as possible without any overhead
All of the components this page is composed of are simple presentational components using styled-components for styling. Adding styled-components to a Next.js project is rather simple and it’s well explained here
I use and the useAmp hook for AMP-enabled images when building for AMP which Next.js supports out of the box
Next Image will not work there since AMP has really strict constraints on what can be used, and we can’t also use next/image yet since we can’t possible list out all possible remote image domains for our jobs since they come from many different sources which could change at any moment. We will explain how we solved this issue in the next blog post.
fetcher is a class which houses all of our data scraping during build time as well as our serverless functions which will be used when users use the search-bar. Here’s how it looks like:

What we basically do above is pretty simple — we import all feed functions, loop through them and call each one of them with a searchables and filter parameters. Searchables is basically an environment variable for search terms which I will always use for different job sites like www.remotefrontendjobs.com and www.remotebackendjobs.com. So for the first one I’d pass stuff like frontend,js,javascript,angular,react and etc. On the other hand, filter will be used when someone uses the search-bar and wants to search for something specific. For the filtering, I use a really neat fuzzy search library called fuse.js. In the end, we also filter all jobs which was posted in the last 20 days, so we don’t clutter our board with old jobs.

We also export a simple serverless function that uses the same fetcher which will be utilized for our runtime search. This function will be automatically served for us in development by using next dev or next start but it will also be deployed into multiple regions if used in Vercel. This only proves that the experience of using Next.js is simply magical 🥰.

For every job board, I use the same feed function to fetch specific data segments of a remote job

So whenever I want to introduce a new job board to the mix I just implement this interface and then I let Next and Vercel (in production) handle filling up my pages with data whenever I build the site.
For example, the feed function for Stackoverflow looks like this:

Here we use rss-parser to fetch the public Stackoverflow jobs rss feed, transform it into a nice data model and return it to our page.

In the initial version of the project I had 7 different job board feed functions exported like this

You can see how easy it is to introduce a new job board to the aggregated mix now.

Recap and what comes next

So let’s recap and take a look at what we have right now.

On every build I will create a page with the first 25 jobs found from an aggregated list of data I retrieve from multiple websites.
I have a simple component which calls into the exported api.ts serverless function, updates the state of and displays the job offers. We also have an infinite scrolling functionality which does the same API call to load more jobs.
The serverless API call hooks into the same process of getting jobs data as during the actual build.

I guess you can see that we have a couple of issues here.

The first set of problems are related to the way we retrieve jobs for the static site generation. How do we update those 25 jobs on our static page? After all, I want to provide my site’s visitors with fresh new jobs as they come in. Do I make a new build everytime I want to update the jobs? And if I do, how do I know when? Would that be expensive? Vercel has a pretty nice free quota, but still if this projects would grow, maybe I will hit it.

Then we have a different set of issues related to the runtime experience on our page. Right now when a user searches or loads more jobs, I go through the whole scraping process all over again. This means literally scrapping all feeds and extracting data from them. This is obviously not ideal since it’s generally a slow process and while one might think it can be solved easily with cache, then we arrive at a whole new set of cache-related problems, like — when to invalidate the cache. 😵

I have solved the first set of problems from above by utilising Incremental Static Regeneration to regenerate the home page at a specific time interval, but only when it’s used. The second issues required a smarter, more scalable approach which would not only let me provide a faster experience to my users, but also a smarter and more fine-grained one. That required introducing my own database (mysql) by using a really cool and well-adopted now ORM called Prisma. Where I am hosting the database and how I am managing and updating it will come with my next article.

This blogpost only covered the initial stages of the remote job aggregator and my next blog posts will build on top of that so you get a clear picture on what decisions were made in terms of performance and UX optimisations and how all of that was achieved. The application right now doesn’t look anything like what was shown above both in terms of design and functionality, and you can take a look at the latest versions at www.remotefrontendjobs.com and www.remotebackendjobs.com. You can use these to find the perfect remote job for you and subscribe for weekly new jobs emails (you can specify that you want jobs with salaries only).

Improve your Angular app performance by using this simple Observable cache decorator 🎉

Angel Nikolov — Mon, 15 Jul 2019 12:21:04 +0000

When we were about to finish development of our applications in SwiftViews we noticed a pattern in all our data-fetching user flows. In spite of the fact that the apps are all data-driven and looked really dynamic, what really was changing in the same user session wasn’t that much, but we were making http requests for new content regardless.
The easiest solution was caching
Caching where? On the server? We already have that, but this doesn’t stop all our apps hitting our services and thus — increasing the load on them.
Maybe we could use service worker since it allows caching API calls?
Yes, that was one of the options since it already has a pretty nice integration with Angular and allowed for a simple solution to selectively cache resources and APIs. However, what we wanted is to be able to not only choose what to cache, but also when to do it.
Just to give you a quick example — we basically wanted to cache all API call results for the page below, but only if we were certain that the data source did not change in some way.

Since this is my personal inventory page, the only way it can actually change is if I add an item from this platform (currently the only way to do so), so I have the information about when this page will change and how long I can return cached content for.
We figured the best way to apply this caching in a selective and configurable way will be to use what the platform already gives us
Since we are using Angular and Typescript ❤️, and all our data calls go through RxJs, we figured that we can create a Cache Observable decorator, which we’d use to only give the caching power to certain methods.
So, before, this was our method which called our server to get the products on the page above:

and that became:

Notice that the @Cacheable() has been applied just to the method we wanted and was also passed cacheBusterObserver which is basically our Subject-based mechanism to tell this exact decorated method to relieve all its caches, when any value is emitted in that stream.
For example, the method below will “cache-bust” the caches of the method above, if the Observable it returns emits a value 😵

This might be the most complex example we have but we also have other configurations implemented like maxCacheCount, maxAge, slidingExpiration and custom cache deciders and resolvers which give us a fine-grain control on what, when and how to cache. See those in the README file.
In the two gifs below, you can see the difference between our app’s performance without and with caching.

https://www.loom.com/share/71a03e4cd478407fa53f131fc112e09a
No Cacheable
The cacheable decorator is not applied yet, so every page load will actually fetch data from the server

https://www.loom.com/share/380a1fa93a9c42d0a5adaa987cd51efb
Cacheable decorator applied 🎉
The cacheable decorator is applied and we can see that all consecutive page loads are faster. Also the user profile page loads instantly, because we have already called the /user/{id}endpoint on the inventory page. Also, the cache busts after we add a new item so our user gets fresh data
So, to sum up, this simple decorator allowed us to:
Selectively cache observable methods (not just endpoints, but maybe also computation-heavy calculations in streams)
Improve the performance of our app, without any business logic refactoring (thanks TS decorators ❤️)
Greatly reduce the load on our servers
If you want to use this decorator in your project, just install it from NPM!
npm install ngx-cacheable
If you have any questions about it or want to contribute, don’t mind opening a pull request @github or commenting below.
Also, if you liked the application, please register and build your own inventory @ swiftviews.
Thank you! 🐦

DEV Community: Angel Nikolov

Anatomy of a fast Next.js job board