Anatomy of a fast Next.js job board

Anatomy of a fast Next.js job board

Hello everyone! I’ve been working remotely for quite some time now and I also have a couple of failed side-projects behind me. Recently, I realised that I should probably invest time in new projects only if they made practical sense for me.
Then it hit me that the very thing that made it possible for me to work on side-projects was actually the thing which made sense for me to optimise and put more effort and ideas in. And that was remote work. One issue with remote work kept coming at me and that was — finding my new work place. I’ve used many different job boards, but all of them had different filters, job selection, mailing lists and etc. which obviously wasn’t ideal.

That’s when I figured that maybe it will be nice to have all of the most used job boards aggregated and presented to users in a nice, fast and efficient way. I purchased several domains which I used to further refine the choice of the user by job categories.
https://www.remotefrontendjobs.com and https://www.remotebackendjobs.com are now listing thousands of jobs, aggregated from more than 14 different sources (all linking back). People could also subscribe to weekly tailored notifications for new jobs (they can also specify if they are only interested in jobs with salaries specified).

In this blog post I will go through how I created a fast, beautiful and tailored job-hunting experience running on Next.js and Vercel ❤️.

To keep you interested, these are the tools that I’ve used for the first version of my job board which I will go through in this article.

When I started out, I wanted to pick the set of tools which would enable me to create a fully functioning job board in a quick and predictable manner. Initially, this was a really quick and neat idea in my head which would’ve taken me no more than a couple of hours. Basically, what I had in mind was a simple page with a list of jobs and a search bar on it.

So what I started with was a simple SSG (Static site generation getStaticProps) page in Next.js and a couple of Serverless functions which I can use to get the actual jobs. Whenever a user would navigate to my page, they would load the list below, which would be generated at build time by scraping some job boards.

There are a couple of important things to note in the code above.

• I am using GetStaticProps to render this page at build time. This basically renders the whole application on the server and outputs a plain old and heavily optimized HTML pages which should load as fast as possible without any overhead

• All of the components this page is composed of are simple presentational components using styled-components for styling. Adding styled-components to a Next.js project is rather simple and it’s well explained here

• I use and the useAmp hook for AMP-enabled images when building for AMP which Next.js supports out of the box

• Next Image will not work there since AMP has really strict constraints on what can be used, and we can’t also use next/image yet since we can’t possible list out all possible remote image domains for our jobs since they come from many different sources which could change at any moment. We will explain how we solved this issue in the next blog post.

• fetcher is a class which houses all of our data scraping during build time as well as our serverless functions which will be used when users use the search-bar. Here’s how it looks like:
  

  What we basically do above is pretty simple — we import all feed functions, loop through them and call each one of them with a searchables and filter parameters. Searchables is basically an environment variable for search terms which I will always use for different job sites like www.remotefrontendjobs.com and www.remotebackendjobs.com. So for the first one I’d pass stuff like frontend,js,javascript,angular,react and etc. On the other hand, filter will be used when someone uses the search-bar and wants to search for something specific. For the filtering, I use a really neat fuzzy search library called fuse.js. In the end, we also filter all jobs which was posted in the last 20 days, so we don’t clutter our board with old jobs.

We also export a simple serverless function that uses the same fetcher which will be utilized for our runtime search. This function will be automatically served for us in development by using next dev or next start but it will also be deployed into multiple regions if used in Vercel. This only proves that the experience of using Next.js is simply magical 🥰.

For every job board, I use the same feed function to fetch specific data segments of a remote job

So whenever I want to introduce a new job board to the mix I just implement this interface and then I let Next and Vercel (in production) handle filling up my pages with data whenever I build the site.
For example, the feed function for Stackoverflow looks like this:

Here we use rss-parser to fetch the public Stackoverflow jobs rss feed, transform it into a nice data model and return it to our page.

In the initial version of the project I had 7 different job board feed functions exported like this

You can see how easy it is to introduce a new job board to the aggregated mix now.

Recap and what comes next

So let’s recap and take a look at what we have right now.

• On every build I will create a page with the first 25 jobs found from an aggregated list of data I retrieve from multiple websites.

• I have a simple component which calls into the exported api.ts serverless function, updates the state of and displays the job offers. We also have an infinite scrolling functionality which does the same API call to load more jobs.

• The serverless API call hooks into the same process of getting jobs data as during the actual build.

I guess you can see that we have a couple of issues here.

The first set of problems are related to the way we retrieve jobs for the static site generation. How do we update those 25 jobs on our static page? After all, I want to provide my site’s visitors with fresh new jobs as they come in. Do I make a new build everytime I want to update the jobs? And if I do, how do I know when? Would that be expensive? Vercel has a pretty nice free quota, but still if this projects would grow, maybe I will hit it.

Then we have a different set of issues related to the runtime experience on our page. Right now when a user searches or loads more jobs, I go through the whole scraping process all over again. This means literally scrapping all feeds and extracting data from them. This is obviously not ideal since it’s generally a slow process and while one might think it can be solved easily with cache, then we arrive at a whole new set of cache-related problems, like — when to invalidate the cache. 😵

I have solved the first set of problems from above by utilising Incremental Static Regeneration to regenerate the home page at a specific time interval, but only when it’s used. The second issues required a smarter, more scalable approach which would not only let me provide a faster experience to my users, but also a smarter and more fine-grained one. That required introducing my own database (mysql) by using a really cool and well-adopted now ORM called Prisma. Where I am hosting the database and how I am managing and updating it will come with my next article.

This blogpost only covered the initial stages of the remote job aggregator and my next blog posts will build on top of that so you get a clear picture on what decisions were made in terms of performance and UX optimisations and how all of that was achieved. The application right now doesn’t look anything like what was shown above both in terms of design and functionality, and you can take a look at the latest versions at www.remotefrontendjobs.com and www.remotebackendjobs.com. You can use these to find the perfect remote job for you and subscribe for weekly new jobs emails (you can specify that you want jobs with salaries only).