Web scraper in Nuxt 3 - part IV - Going live on Netlify

Welcome back to my tutorial about web scraping with Nuxt 3. This is is the last part finishing the mini-series:

1. Introduction and setting up
2. Backend scraping service in Nuxt
3. Displaying the results on frontend
4. Going live on Netlify

We left our demo Nuxt application scraping data from JEP Index website and also displaying them in own home page. Now the last thing remains to be done - publish the result.

Nuxt deployment options

I have been playing with building websites for at least 20 years. I remember back in my school days when my “stack” was vanilla PHP randomly glued inside HTML. It wasn’t that easy to get the local PHP server and MySQL database up and running. And then managing to find a hosting platform to go live with your site. It wasn’t the hardest thing in the world, but still challenging. If someone tries to tell you “The old times were better”, don’t listen. They weren’t. Nowadays stuff is much easier to do.

This also counts for deploying web applications. There are vast options. The way I’ll be showing in this tutorial is not the only one, possibly not even the best one. Basically, any way of obtaining a working server with JavaScript runtime will allow you to run your Nuxt app. And there is a growing number of cloud providers capable of running Nuxt preconfigured almost out-of-the-box.

When I was first thinking about the possibilities, I’ve decided to try deploying to Netlify first. And I was left amazed by how easy it was. I didn’t even seek other solutions because this was totally what I wanted. Let me show you.

Netlify

There is much more in the whole Netlify platform, but we’ll be focusing on the automated cloud deployment feature.

The first appealing thing is that the service is available for free. Naturally the free tier has limits (100 GB bandwidth and max. 300 build minutes per month), but you're not likely to hit this with demo projects anytime soon, so you are free to experiment and learn, and shall your project be successful one day, you may think of upgrading to much better plan for $19 per month, which is not that much. Or you will decide to move somewhere else, but not before you can gain insight into Nuxt development without worrying much about the deployment. You can run multiple sites under free account as well, you're just limited with one build at the time, which is also barely an issue.

The second interesting concept I absolutely love about this service is that you practically don’t have to prepare your Nuxt project for Netlify deployment. Once it runs locally, it will run on Netlify.

Checking the build locally

To make sure there is no hidden gotcha, turn off your dev server (previously started by npm run dev) and let Nuxt build itself with npm run build command. Once your build is finished, try npm run start. You should see something like this:

Visiting localhost:3000 in your browser should show you the very same and working page as before (just not capable of hot module reloading as the build is pinned to certain source code version).

If this works for you, you may proceed with deploying. If you're experiencing some problems, don't hessitate to ask for help in the comments.

Deploying with GitHub

Netlify is so user friendly, that you could even just drag’n’drop the output files and the platform will build everything around it. But this is a little too barbaric if you ask me. Thus, I would like to suggest you initiate a Git repository on your project and publish it on GitHub (or some alternative platform, GitLab, BitBucket, etc.), if you haven’t already. I won’t be showing you how to do it here. If needed, there are plenty of good tutorials already, namely you can choose to follow GitHub docs.

Doing this we can now tell Netlify to connect to our public repository and use it as a source for its build and deployment. Netlify then fetches the sources, performs the build, deploys the output container into its infrastructure and makes it available to the internet. The automated process will then run every time you push something new into your main branch (by default, you can change it, or even configure parallel builds, but that’s out of the scope, check the docs if you’re interested).

From now on, I will only be showing up the GitHub setup. The others should be quite the same, but there might be some differences.

Step by step deployment guide

1) Create an account

By default, you can just simply connect your GitHub account. Click the button, authorize the request, and proceed.

2) Start on “Team overview page” and select “Add new site” – “Import an existing project”

3) Let’s “Deploy with GitHub”

4) Pick up your repository

Here is a little gotcha. Although you’ve already authorized Netlify to GitHub, by default it still won’t disclose your repositories. You need to explicitly allow them. Go to the bottom of the page and click the link:

The app will open in a new window and will probably require authorization again. I have my GitHub 2FA guarded with the mobile app, so I always have to open it and re-type the numbers it shows me:

Alternatively, it is also possible to get there via GitHub account – visit “Settings” and then pick “Applications” in the “Integrations” section. There will be Netlify app you can configure:

The important section is at the bottom – “Repository access”:

Here you can configure which of your repositories would Netlify see. Pick the one you want to deploy (or all of them to save time in the future, but I’d advise against that), save the selection and go back to Netlify deployment.

5) Configure the automated build

After picking the now available repository, we advance to a screen with final configuration:

A form follows that allows you to make changes.

• You can pick the “Branch to deploy”, but we don’t have any other than main on our project anyway.
• “Base directory” should be left empty. It allows you to potentially run Nuxt instance elsewhere than in the root.
• “Build command” is a npm script that will be auto executed during the build phase. The default value of npm run build would probably be enough for your first demo, but I tend to add npm ci && npm run build. The npm-ci command is an alternative version of npm install tailored for automated builds. It will prolong the build time for a bit, but it will also guarantee everything always initializes smoothly and correctly. Be aware you need to include package-lock.json into Git repo, otherwise the ci command will fail.

Update May 2024: I recently switched to pnpm package manager in most of my project, where the command combo should be pnpm install --frozen-lockfile && pnpm run build doing roughly the same (you'll need pnpm-lock.yaml lock file).

• “Publish directory” – here an adjustment is required, as the default value .nust/dist does not work well for some reason. Change it to dist only.
• “Functions directory” is not going to be used by our deployment. We also don’t need to “Add environmental variables” right now. Once you will create some more advanced Nuxt applications using their runtime config, this place will become interesting for you.

Rest assured you can always go back to the settings and change them as you’ll see fit. For now, we are good to go with this:

Take a deep breath and hit the “Deploy” button.

Deployment process and history

After a short while a new page appears. The initial build was started:

Once the build finished, site become available by clicking on the green link:

You might be scared of the random and ugly subdomain name but worry not! Just click “Site configuration” button and then “Change site name”. You will be able to pick something more suitable – like jepeek.netlify.app, but I already took this one, so you will have to be more creative.

It is also quite easy to configure your fully custom domain instead of being forced to use a subdomain under netlify.app. For example, my blog (only in Czech language) runs on Netlify but is available via https://master-coda.cz/ which is the domain I bought here in Czechia and just had to provide necessary DNS settings to link it with my deployment. Again, this is outside the scope of this lesson, but here are the docs to help you, in case you’d like to try it.

We are about done. Unless you ran into some trouble (feel free to express them and ask in the comments section), your site is deployed and will now being deployed automatically every time you push a new code into your main branch in linked GitHub repository.

You can check this on “Site overview” in the “Production deploys” section:

Each record links to the respective Git commit message from your repo and you can see the details of the deployment process including the logs. This might become useful if there are some problems with the build. The retention of those data is limited to the last 4, but mostly you would look for the newest one anyway.

Conclusion

So, this is it! During four parts of this tutorial, we have been exploring the Nuxt framework, building a web scraper with it and learned how to deploy our finished application to a cloud hosting service. You saw how many things are just plain and easy when using Nuxt (and Netlify).

I hope it was interesting for you and you have learned something new. If you are already thinking about how to use any of this information in your future projects, I wish you the best of luck with them. And if you become Nuxt-addict like me, we might meet each other once again soon.