DEV Community: Shane Jennings

Creating a Project Using Public Data for Fun and Profit: Part 4

Shane Jennings — Fri, 05 May 2023 17:30:39 +0000

This blog is part 4 of "Creating a Project Using Public Data for Fun and Profit".

Previous posts:

In this part, we will cover:

Adding support for Twitter and Facebook cards
Measuring website performance using Google Analytics
Measuring SEO using Google Search Console
Monitising our website using Google AdSense

All the code referenced in this post can be found in my repo here. As well, you can find a live version of the application we build here.

Adding Social Media Support

Adding social media support is a great way to improve the appearance of your website when shared on social media platforms. The tags provided in the sections below will add a card preview on Twitter and Facebook, which will contain a title, description, and image.

To add Twitter support, add the following tags to _app.tsx in <Head>:

<meta name="twitter:card" content="summary" />
<meta name="twitter:url" content="https://public-data-demo.vercel.app" />
<meta name="twitter:title" content="Best Driving Test Pass Rates Near Me" />
<meta name="twitter:description" content="Give yourself the best opportunity to pass your driving test. Find the driving test centre that has the best pass rate near you. Find in locations such as Manchester, London, Birmingham, Newcastle, Leeds, Wales, Scotland, anywhere in the UK." />
<meta name="twitter:image" content="https://public-data-demo.vercel.app/icon-512x512.png" />
<meta name="twitter:creator" content="@shanepjennings" />

Make sure to change the URL, image, and creator tags to reflect your own!

Using a link to your website in a tweet will now show a card, like this:

To add Facebook support, add the following tags to _app.tsx in <Head>:

<meta property="og:type" content="website" />
<meta property="og:title" content="Best Driving Test Pass Rates Near Me" />
<meta property="og:description" content="Give yourself the best opportunity to pass your driving test. Find the driving test centre that has the best pass rate near you. Find in locations such as Manchester, London, Birmingham, Newcastle, Leeds, Wales, Scotland, anywhere in the UK." />
<meta property="og:url" content="https://public-data-demo.vercel.app" />
<meta property="og:image" content="https://public-data-demo.vercel.app/icon-512x512.png" />

Again, make sure to change the URL and image tags to reflect your own!

Using a link to your website in a Facebook post will now show a card, like this:

To test how your website will appear on various social media platforms from various devices, you can use the Open Graph Simulator. Simply enter your website URL and you'll be shown how your website would appear on various platforms.

Performance

Your application is now live, great! But now you might be wondering, "How can I see if anyone is even using my website?".

This is where Google Analytics comes in. It's a powerful tool that allows you to measure your website's performance, track user behaviour, and gain insights that can help you improve your site.

Here we'll walk through how to get started with Google Analytics.

Step 1: Create an Account and Property

First, go to analytics.google.com and click the "Start measuring" button. This will prompt you to create an account and a property. If you're a single developer, you'll likely have one account to encompass all of your websites and a property per website.

Once you've created your property, you should be given a Measurement ID. If not, click on the Gear icon on the bottom left of the screen, click "Data Streams" under the property column, and select the "Web" tab.

Here, you'll find the Measurement ID.

Step 2: Add the Google Analytics Script to Your Website

To enable analytics collections for your website, you'll need to add the Google Analytics script. Go to _app.tsx and add the following code before the <Head> element:

<Script id="google-tag-manager" strategy="lazyOnload"
        src="https://www.googletagmanager.com/gtag/js?id=G-XXXXXX"/>
<Script id="google-analytics" strategy="lazyOnload">
  {`
                window.dataLayer = window.dataLayer || [];
                function gtag(){dataLayer.push(arguments);}
                gtag('js', new Date());

                gtag('config', 'G-XXXXXX', {
                    page_path: window.location.pathname,
                    }); 
            `}

</Script>

Make sure to replace the G-XXXXXX with your own Measurement ID.

Step 3: Start Tracking Your Website's Performance

Once you've added the Google Analytics script to your website, you can start tracking its performance. Visit your website a few times and click around on various pages. Eventually, you'll start seeing data in the Google Analytics console.

Google Analytics tracks a lot of useful information, such as the number of users who have visited your website, where they're from, which pages are the most popular, and much more. I recommend taking some time to explore the different reports and analytics to gain valuable insights that can help you improve your website.

And the best part? Google Analytics is free to use!

SEO

I've mentioned SEO a few times in this post, however, if you haven't heard of it before it stands for Search Engine Optimisation, and it's all about getting your website to show up in search engine results, like Google. The higher your website ranks in search results, the more likely people are to click on it. And if you're running a monetized website, good SEO can mean more income.

Next.js has an excellent introduction to SEO, which you can find here.

There are plenty of SEO tools out there, some free and some not. Some of these tools can help with keyword research, which can help you figure out which words to use on your website to improve your chances of showing up in search results. But in this section, we'll be focusing on Google Search Console, a free tool that I find particularly useful for SEO.

Google Search Console has two features that I find particularly helpful:

Indexing : You can see if Google has indexed pages from your website. Pages need to be indexed to show up in a Google search.
Performance : You can see how many times your website has appeared in Google searches, how many clicks it's received, what people searched for to find it, and more.

Step 1: Sign Up With Google Search Console

To get started with Google Search Console, go to search.google.com/search-console and enter your website's domain under the "Domain" box on the left. Then click "Continue."

Step 2: Verify Domain Ownership

You'll see a screen asking you to verify domain ownership, like this:

We need to complete this step before we can use Google Search Console. Unfortunately, we can't do this with the basic Vercel domain that we're using for free.

This step requires you to have purchased a domain in the previous step. The steps to verify domain ownership will vary depending on your domain provider.

If you purchased your domain through Google Domains, you can verify ownership by:

Opening your domain in Google Domains
Select "DNS" from the left-hand sidebar
Clicking "Manage custom records"
Clicking "Create new record"

Once you've created the custom record, click the "Verify" button in Google Search Console. It may take some time for the change to take effect, so you may need to check back later.

Indexing

Now, you might be wondering why you're not showing up on Google's search results yet. Well, the answer is simple: Google hasn't indexed your website yet.

First, let's talk about what indexing is. Google uses a "crawler" to find your website's robots.txt file. This file tells the crawler which URLs it can access to crawl. From there, the sitemap.xml file, which contains the locations of all of your static pages, is found. The crawler will then start indexing each of these pages.

It is important to note that this process can take months , especially if the website isn't appearing in a lot of Google search results. But there are ways to speed it up.

One way is to check if Google has found your sitemap.xml file. You can see this by clicking on "Sitemaps" on the left-hand sidebar. If your sitemap has been found then it will be listed there.

If your sitemap hasn't been found, you can speed up the process by submitting a link to your sitemap using the "Add a new sitemap" box. This will add the sitemap to the crawler's queue, increasing the likelihood that your website will be indexed sooner.

Once your sitemap has been discovered, Google will automatically start discovering pages on your website. In the screenshot above you'll see under the "Discovered pages" heading that 67 pages have been discovered on my website, drivingpassrate.co.uk.

Once you have some pages appearing you can dig into this deeper by clicking on "Pages" on the left-hand sidebar. This will show you which pages have been indexed and which have yet to be indexed.

In the image above you can see how pages on my website have been indexed over time.

At the bottom of this page, we can see more details on why some pages have not been indexed.

Here we can see that Google has:

discovered 15 pages (by using the sitemap) but has not yet crawled or indexed them
has crawled 4 pages but has not yet indexed them.

We can drill deeper again to see which pages haven't been indexed.

I mentioned that this process can take months, and you can see proof of that in the image above, where drivingpassrate.co.uk/pass-rates/derby was crawled in August last year but still hasn't been indexed.

We can speed up this process by forcing our pages into Google's indexing queue. To do this, search for your page in the search bar at the top that says "Inspect any URL". Note that you need to search for the whole URL, including HTTPS. The search is also case-sensitive so make sure that is also correct.

Once you've searched for a page you can check that that page exists by clicking the "Test Live URL" button. If you've entered the wrong URL then you'll see "Page cannot be indexed: Not found (404)", like this:

If you've entered the correct URL then you should see "Page can be indexed", like this:

To force the page into the indexing queue click the "Request Indexing" button.

You should then see a popup stating that the URL was added to the queue. Note that it also says that submitting the same page multiple times will not change the queue position, so don't bother spamming the "Request Indexing" button!

Keep in mind that even though your page has been added to the queue, it can still take a long time to be indexed. However, it will be faster than waiting for the crawler to automatically crawl it. Also, note that you can only request a small number of pages to be indexed at one time before your quota is exceeded (around 10 pages per day). So, be strategic in which pages you request to be indexed.

If you exceed this quota, you'll see this popup when trying to request indexing:

Performance

With this feature, you can see how many people have seen your results when searching in Google, how many people have clicked on your results, and what people were searching for when they saw your results.

To access this feature, simply click on "Performance" on the left-hand sidebar of the Search Console dashboard. You'll be presented with a graph that displays several metrics, including total clicks, total impressions, average click-through rate (CTR), and average position.

Scrolling down, you'll find a table that displays more detailed information, such as the queries people searched for, the pages that appeared in the search results, the countries people were searching from, the devices they used to search, and the dates your pages appeared in search results. Each result set displays the number of impressions and the position of your website in the search results.

To dive even deeper, you can click on any of these results to see the page that appeared in the search results, as well as other metrics such as country and device. With this information, you can better understand how your website is performing in search results and make improvements as needed.

For example, by clicking the top result "winchester pass rate" we can see the page that appeared in the search results, as well as all the other metrics such as country and device.

Keyword Research

When it comes to SEO, it's also important to consider keyword research. While Search Console can show you what keywords people searched for when your website appeared in search results, it can't tell you what similar keywords people searched for when your website did not appear in the results. This is where keyword research comes in.

There are free tools available, such as Ahrefs Keyword Generator, that can help you conduct keyword research. By typing in a number of keywords related to your website, you can find out what people are searching for related to those keywords. For example, if you search for the term "driving test centre pass rate," you might find that people are searching specifically for "Mill Hill driving test centre pass rate," with around 100 searches per month.

Armed with this information, you can optimize your website's content to target these keywords and improve your SEO.

Monitise your Website

There are numerous ways that you can earn income from a website, but one popular method is by adding ads. And Google AdSense is an easy and effective way to start earning some revenue.

Step 1: Sign Up with Google AdSense

To get started, simply head over to adsense.google.com/start/ and sign up. Once you have created an account, you can add your website to start displaying ads.

Step 2: Add the Script

After signing in, navigate to the "Sites" tab on the left sidebar, and click on "New site" to add your website. Once you've done that, you will see the code you need to add to your website to start showing ads.

To add the code to your website, simply go to _app.tsx and add the following code just before the <Head> element:

<Script
src="https://pagead2.googlesyndication.com/pagead/js/adsbygoogle.js?client=ca-pub-XXXXX"
crossOrigin="anonymous"/>

Make sure to replace "ca-pub-XXXXX" with your own unique AdSense publisher ID.

Step 3: Enable Ads

After adding the code to your website, go back to the Google AdSense window and tick the "I've placed the code" option and click "Next." Next, click "Request review", so that your website will be reviewed for eligibility.

While waiting for the review process, you can decide on your ad placement. With Google AdSense, you have the option to let the system automatically place ads on your site or choose ad placements manually. In this post, we'll focus on the former option.

Navigate to the "Ads" tab on the left sidebar and click the pencil icon next to your website to edit its settings.

From there, select the "Auto ads" option to turn on automatic ad placement. Sometimes, the ads might not show up immediately, so try clicking "Apply to site" and coming back later to see if the ads have appeared. You should see some ad placeholders added to your site.

If you're not satisfied with the ad placement, you can always choose to manually place them on your site. Check out this guide to learn how.

It's important to note that ads won't appear immediately after adding the code to your website. Google AdSense will review your website to ensure that it meets its policies, which can take up to two weeks.

Conclusion

And there we have it! Firstly, well done for getting through this whole process. Building an entire application and ensuring it's production-ready is no easy feat, so congratulations on getting through this entire guide! Let's cover what we've done:

Built a website using public data
Added a sitemap and robots.txt for SEO
Added PWA support
Deployed the website using Vercel
Added our own domain name
Added our own email address for our domain
Added support for Twitter and Facebook cards
Measured website performance using Google Analytics
Measured SEO using Google Search Console
Monitised the website using Google AdSense.

That is a lot!

You can find all the code used in this post in my repo here. As well, you can find a live version of the application we built here.

Till next time!

Creating a Project Using Public Data for Fun and Profit: Part 3

Shane Jennings — Tue, 02 May 2023 15:37:20 +0000

This blog is part 3 of the "Creating a Project Using Public Data for Fun and Profit" series.

Previous posts:

In this part, we will cover:

Adding a favicon
Adding a sitemap and robots.txt
Adding PWA support
Deploying our website
Setting up a domain name
Setting up email forwarding

All the code referenced in this post can be found in my repo here. As well, you can find a live version of the application we build here.

Adding the Finishing Touches

Congratulations on building your application! If you're planning on deploying it to the public, there are a few finishing touches that can take it to the next level of professionalism. In this section, we'll go over some easy but important tweaks that will make your app look even better.

Adding an Icon

When you open your application at http://localhost:3000, you may notice that the favicon (the small icon that appears on the tab in your browser) is set to the Next.js logo. That's not very personalized, is it? We will change this to be the icon for our website.

There are plenty of places online where you can get free assets. One of my favorites is flaticon.com. Let's use the icon here for our website.

Before we get started, remember that favicons have a specific size: 32x32 pixels. So make sure to download the icon as a 32x32 PNG file. Once you have it, you'll need to convert it to a .ico file, which is what we need for our favicon. There are plenty of online tools for doing this, but one option is here. Just upload the PNG file and convert it to .ico.

Now that you have your favicon.ico file, it's time to replace the existing one in the public directory. This will ensure that your new icon is used when your app is deployed.

Once you've completed this step, head back to http://localhost:3000. You should see that the icon has changed on the tab.

Adding a Sitemap and robots.txt

Your website is almost ready to go live! However, before we deploy it, let's add a sitemap and a robots.txt file to improve its search engine optimization (SEO).

Sitemaps help search engines understand the structure of your website and find all the relevant pages.

The robots.txt tells search engines how to crawl your website.

To add these files to your website, we can use a package called next-sitemap. This package will automatically generate a sitemap and robots.txt for all of our static pages.

Step 1: Add next-sitemap.config.js

First, we need to create a next-sitemap.config.js file in the root directory of our project. Copy the following code into the file:

/** @type {import('next-sitemap').IConfig} */
const config = {
    siteUrl: process.env.SITE_URL || 'https://yourdomainname.com',
    generateRobotsTxt: true, // (optional)
    // ...other options
}
module.exports = config

This sets up the configuration for next-sitemap. Notice that siteUrl is set to https://yourdomainname.com. Don't worry, we will update this later.

Step 2: Install next-sitemap

Next, let's add the next-sitemap package to our project by running the following command in our terminal:

npm install next-sitemap

Step 3: Add postbuild Step

Once the package is installed, we need to add a postbuild step to our package.json file to generate the sitemap. Open the package.json file and add the following line after the build command:

    "build": "next build",
    "postbuild": "next-sitemap",

Now, we can generate the sitemap by running the following command in our terminal:

npm run build

Make sure to have stopped your instance of npm run dev first.

Once the build is complete, you should see that 3 files have been generated in your public folder.

robots.txt:

# *
User-agent: *
Allow: /

# Host
Host: https://yourdomainname.com

# Sitemaps
Sitemap: https://yourdomainname.com/sitemap.xml

sitemap.xml:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap><loc>https://yourdomainname.com/sitemap-0.xml</loc></sitemap>
</sitemapindex>

sitemap-0.xml:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
  <url><loc>https://yourdomainname.com</loc><lastmod>2023-04-06T12:36:31.039Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
  <url><loc>https://yourdomainname.com/cities</loc><lastmod>2023-04-06T12:36:31.039Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
  <url><loc>https://yourdomainname.com/pass-rates</loc><lastmod>2023-04-06T12:36:31.039Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
  <url><loc>https://yourdomainname.com/pass-rates/aberdeen</loc><lastmod>2023-04-06T12:36:31.039Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
  <url><loc>https://yourdomainname.com/pass-rates/bangor</loc><lastmod>2023-04-06T12:36:31.039Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
  <url><loc>https://yourdomainname.com/pass-rates/bath</loc><lastmod>2023-04-06T12:36:31.039Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
  <url><loc>https://yourdomainname.com/pass-rates/birmingham</loc><lastmod>2023-04-06T12:36:31.039Z</lastmod><changefreq>daily</changefreq><priority>0.7</priority></url>
...

How to Make Your App a Progressive Web App (PWA)

PWAs are a game-changer for providing a consistent experience across a variety of devices, and they allow users to install your web app on their device as if it were a native Android or iOS app. And the good news is that with Next.js, setting up your application as a PWA is quick and easy!

You can read more about them here.

Step 1: Install next-pwa

The first thing we'll do is install next-pwa with the following command:

npm install next-pwa

Step 2: Update next.config.js

Next, we need to update our next.config.js file to use the next-pwa plugin. Here's the code you'll need:

/** @type {import('next').NextConfig} */
const withPWA = require('next-pwa')({
  dest: 'public'
})

const nextConfig = withPWA({
  reactStrictMode: true,
  swcMinify: true,
})

module.exports = nextConfig

Step 3: Create manifest.json

To make our PWA work, we need to create a manifest.json file. The easiest way to do this is to use a generator like simicart. Make sure you download the icon at size 512 here as you'll need that for the generator.

Add the following properties to the Simicart manifest generator:

Click "Generate Manifest" and you'll download a zip. Extract this zip to your public directory and rename manifest.webmanifest to manifest.json.

Your manifest.json should look like this:

{
  "theme_color": "#008000",
  "background_color": "#FFFFFF",
  "display": "standalone",
  "scope": "/",
  "start_url": "/",
  "name": "Best Driving Test Pass Rates Near Me",
  "short_name": "Driving Test Pass Rates",
  "description": "Give yourself the best opportunity to pass your driving test. Find the driving test centre that has the best pass rate near you. Find in locations such as Manchester, London, Birmingham, Newcastle, Leeds, Wales, Scotland, anywhere in the UK.",
  "icons": [
    {
      "src": "/icon-192x192.png",
      "sizes": "192x192",
      "type": "image/png"
    },
    {
      "src": "/icon-256x256.png",
      "sizes": "256x256",
      "type": "image/png"
    },
    {
      "src": "/icon-384x384.png",
      "sizes": "384x384",
      "type": "image/png"
    },
    {
      "src": "/icon-512x512.png",
      "sizes": "512x512",
      "type": "image/png"
    }
  ]
}

Step 4: Add PWA tags to the Head element

Next, we need to add more tags to our <Head> element in the _app.ts file to provide more information on our PWA for different devices. Replace the <Head> element with the following:

<Head>
  <title>Best Driving Test Pass Rates Near Me</title>
  <meta name="description"
        content="Give yourself the best opportunity to pass your driving test. Find the driving test centre that has the best pass rate near you. Find in locations such as Manchester, London, Birmingham, Newcastle, Leeds, Wales, Scotland, anywhere in the UK."/>
  <meta name='application-name' content='Best Driving Test Pass Rates Near Me' />
  <link rel="icon" href="/favicon.ico"/>

  {/* iOS */}
  <meta name="apple-mobile-web-app-capable" content="yes" />
  <meta name="apple-mobile-web-app-status-bar-style" content="default" />
  <meta name="apple-mobile-web-app-title" content="PWA App" />
  <meta name="format-detection" content="telephone=no" />

  {/* Android */}
  <meta name="mobile-web-app-capable" content="yes" />
  <meta name="theme-color" content="#008000" />

  <link rel="manifest" href="/manifest.json" />
  <link rel="shortcut icon" href="/favicon.ico" />
  <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500" />

</Head>

Now, we can generate the PWA config by running the following command in our terminal:

npm run build

You'll see 2 new files in the public directory, sw.js and workbox-<guid>.js.

These files are crucial for PWAs and you can learn more about them here.

Once the build process is complete, start up the app by running npm run start. Then, navigate to http://localhost:3000/. You should now see an option to install the application.

Just a quick heads up: I had to change the display property to standalone in manifest.json before Chrome allowed me to install the app as a PWA. You might need to do the same.

Deploy the Application

We're now ready to deploy our application! We'll be using Vercel to deploy our Next.js application, as it's free for personal projects and is made by the creators of Next.js themselves.

Deploying with Vercel

To deploy your application on Vercel, start by signing up on their website using your GitHub account. Once you're logged in, head to the Dashboard and click the "Add New..." button followed by "Project". From there, click the "Import" button and select your project.

After that, leave all the fields on the next screen as they are and click "Deploy". Vercel will now begin deploying your application, and you'll be able to see the progress of the deployment on your screen.

Once the deployment is finished, you can select your project and preview the landing page of your app. Click the "Visit" button to access your website, which now has a domain associated with it (e.g., public-data-demo.vercel.app).

At this point, you now have a domain. For example, mine is public-data-demo.vercel.app. You can now go and change your next-sitemap.config.js file to point to this domain if you wish. Make sure to re-generate the robots.txt and the sitemap by running npm run build and check this into your repo.

If you would rather get your own domain without the vercel.app suffix then hang on till the next section!

Vercel Analytics

One great feature of Vercel is its analytics. You can enable these for free by clicking the heartbeat button and then clicking "Enable".

This will give you valuable insights into your site's performance and an overall experience score based on multiple factors.

A low score can negatively impact SEO with Google, so it's important to know how your site is performing and make any necessary improvements.

Buy a Domain Name

You may want to have your own domain name that isn't associated with vercel. Domains can vary wildly in price depending on how popular the domain is.

I usually get my domains from Google Domains which is quick and easy to use. Say for example we wanted a domain related to the phrase public-data-demo. We can search for this in Google Domains and buy one for 10 a year:

We won't be walking through how to add a custom domain here, however I can assure you that is it very easy to do with Google Domains and Vercel. You can find a guide on how to do this here.

You can now go and change your file to point to this domain if you wish.

Once your domain has been added make sure to update your next-sitemap.config.js file to reflect your new domain name! Also, make sure to re-generate the robots.txt and the sitemap by running npm run build and push this into your repo. We'll get into why this is important in a later section.

Email Forwarding

If you want to use a custom email address with your domain, but don't want to pay for Google Workspace, you can set up email forwarding for free.

To do this, go to your domain in Google Domains and navigate to the "Email" section. Click "Add email alias" and enter your desired email address at your domain, along with your personal email address that you want emails forwarded to. This will allow you to create an email address with your domain name (e.g., hello@drivingpassrate.co.uk) and forward it to your personal email.

Conclusion

This completes part 3!

Let's review what we've done here:

Added a favicon
Added a sitemap and robots.txt
Made it a PWA
Deployed with Vercel
Got a domain name
Set up email forwarding

You can find all the code used in this post in my repo here. As well, you can find a live version of the application we built here.

Look out for the final part, where we'll be adding social media support, finding out how we can track website performance and SEO, and monetising our website!

Updated: Part 4 is now available. Read it here.

Till next time!

Creating a Project Using Public Data for Fun and Profit: Part 2

Shane Jennings — Wed, 19 Apr 2023 15:22:12 +0000

This blog is part 2 of "Creating a Project Using Public Data for Fun and Profit" series.

Previous posts:

part 1

In this part, we will cover:

Adding the ability to search by postcode and radius on the landing page and the pass rates page
Adding autocomplete to the postcodes search field
Creating a page for each city, showing the nearest test centres
Creating a page listing all cities, and adding a link to this page from the landing page

All the code referenced in this post can be found in my repo here. As well, you can find a live version of the application we build here.

Adding Search Functionality

Now that we have a page that shows the nearest and best test centres based on a postcode, we need to add a user-friendly way to search by postcode.

To do this we'll add a form, with a text field for the postcode, and a dropdown for the radius.

Step 1: Create the Search Component

We'll start by adding a new component under our component directory called Search.tsx. This will contain the elements we just described and a submit button that will navigate the user to our /pass-rates page. Search.tsx should contain the following:

import {Box} from "@mui/system";
import {Button} from "@mui/material";
import Paper from "@mui/material/Paper";
import React, {useEffect, useState} from "react";
import {useRouter} from "next/router";

export default function Search({initialPostcode, initialRadius}:any) {
    const router = useRouter()
    const [postcode, setPostcode] = useState<string | null>(null);
    const [radius, setRadius] = useState<number>(10);

    useEffect(() => {
        setPostcode(initialPostcode);
        setRadius(initialRadius)
    }, [initialPostcode, initialRadius])

    function handleSubmit() {
        router.push(`/pass-rates?postcode=${postcode}&radius=${radius}`);
    }

    return (
        <Paper variant="outlined" sx={{p: 1}}>
            <div style={{display: 'flex', flexFlow: 'row wrap'}}>
                <Box sx={{m: 1, flex: 1}}>
                    <TextField label="Postcode" value={postcode} onChange={e => setPostcode(e.target.value)}/>
                </Box>
                <FormControl sx={{m: 1, minWidth: 120, flex: 1}}>
                    <InputLabel id="demo-simple-select-label">Radius</InputLabel>
                    <Select
                        labelId="demo-simple-select-label"
                        id="demo-simple-select"
                        value={radius}
                        label="Age"
                        onChange={e => setRadius(e.target.value as number)}
                    >
                        <MenuItem value={5}>5 miles</MenuItem>
                        <MenuItem value={10}>10 miles</MenuItem>
                        <MenuItem value={20}>20 miles</MenuItem>
                        <MenuItem value={30}>30 miles</MenuItem>
                    </Select>
                </FormControl>

                <Box sx={{m: 1, mt: 1.8, flex: 1, display: 'flex', flexDirection: 'column'}}>
                    <Button sx={{alignSelf: 'center'}} variant="outlined" onClick={handleSubmit} size="large">Search</Button>
                </Box>

            </div>

        </Paper>
    );
}

Here we can see that we have a TextField element for the postcode and a Select element for the radius. Our submit Button element calls the handleSubmit function when pressed, which navigates the user to the /pass-rates page, passing in the postcode and radius as query parameters.

Step 2: Add Search Component to Landing Page

Next we need to add the <Search> element to our landing page. Open index.tsx and add the following after the <div> containing the <Typography> element:

<Search initialPostcode={''} initialRadius={10}/>

Go to http://localhost:3000. You should now see the new search fields.

Try entering a postcode and clicking search, it will bring you to the /pass-rates page.

Step 3: Add Autocomplete to Postcode Search

Currently, we have no validation for postcodes in our search component. One way that we can validate postcodes is to use postcodes.io again.

We can call postcodes.io to check if the entered postcode is a real postcode. Even better, postcodes.io offers an autocomplete endpoint which we can use to show an autocomplete dropdown.

We'll add the following functions to our PostcodesAPI.ts file:

export async function getPostcodeSuggestions(partial: string) {
  return (await getPostcodeAutocompleteResponse(partial))?.result;
}

const getPostcodeAutocompleteResponse = async (partial: string) : Promise<PostcodeAutocompleteResponseDTO | null> => {
  return await fetch(`${baseUrl}${partial}/autocomplete`, {
    headers: {
      'Content-Type': 'application/json',
      'Accept': 'application/json'
    }
  }).then(response => {
    return response.json();
  }).catch(error => {
    console.error(error);
    return null;
  })
}

export const isValidPostcode = async (postcode: string) : Promise<boolean> => {
  return await fetch(`${baseUrl}${postcode}`, {
    headers: {
      'Content-Type': 'application/json',
      'Accept': 'application/json'
    }
  }).then(response => {
    return response.ok;
  }).catch(error => {
    console.error(error);
    return false;
  })
}

interface PostcodeAutocompleteResponseDTO {
  result: string[]
}

We have added:

getPostcodeSuggestions, which calls Postcodes.io's autocomplete endpoint to get the autocomplete suggestions.
isValidPostcode, which checks that the passed in postcode is a valid postcode

Next, we'll create a new component with our autocomplete functionality, which will replace the postcode TextField element in the Search component.

Firstly, we'll need to install lodash.

Run the following:

npm install lodash
npm install @types/lodash

We then need to create the new component, PostcodeAutocomplete.tsx in the components directory, which should contain the following:

import * as React from 'react';
import {useCallback, useEffect} from 'react';
import Box from '@mui/material/Box';
import TextField from '@mui/material/TextField';
import Autocomplete, {AutocompleteRenderInputParams} from '@mui/material/Autocomplete';
import LocationOnIcon from '@mui/icons-material/LocationOn';
import Grid from '@mui/material/Grid';
import debounce from 'lodash/debounce';
import {getPostcodeSuggestions} from "./PostcodesAPI";
import {CircularProgress} from "@mui/material";

export const getOptionsAsync = (query: string): Promise<string[]> => {
    if(!query) return Promise.resolve([]);
    return getPostcodeSuggestions(query).then(suggestions => suggestions ? suggestions : []);
};

interface PostcodeAutocompleteProps {
    setPostcode: (s: string | null) => void
    postcodeError: boolean
    setPostcodeError: (b: boolean) => void
    submitForm: () => void
    postcode: string | null
}

export default function PostcodeAutocomplete({
                                                 setPostcode,
                                                 postcodeError,
                                                 setPostcodeError,
                                                 submitForm,
                                                 postcode
                                             }: PostcodeAutocompleteProps,) {
    const [options, setOptions] = React.useState<string[]>([]);
    const [value, setValue] = React.useState<string | null>(postcode);
    const [searchQuery, setSearchQuery] = React.useState<string>("");
    const [isLoading, setIsLoading] = React.useState<boolean>(false);

    const getOptionsDelayed = useCallback(
        debounce((query: string, callback: (options: string[]) => void) => {
            setOptions([]);
            getOptionsAsync(query).then(callback);
        }, 300),
        []
    );

    useEffect(() => {
        setValue(postcode);
    }, [postcode])

    useEffect(() => {
        setIsLoading(true);

        getOptionsDelayed(searchQuery, (options: string[]) => {
            setOptions(options);

            setIsLoading(false);
        });
    }, [searchQuery, getOptionsDelayed]);

    const onChange = (event: unknown, value: string | null) => {
        setValue(value);
        setPostcode(value);
        setPostcodeError(false);
    };

    const onInputChange = (event: unknown, value: string) => {
        setSearchQuery(value);
        setPostcode(value);
        setPostcodeError(false);
    };

    const keyPress = (e: React.KeyboardEvent<HTMLDivElement>) => {
        if (e.key == 'Enter') {
            submitForm()
        }
    }

    const renderInput = (
        params: AutocompleteRenderInputParams
    ): React.ReactNode => {
        return (
            <TextField {...params} label="Postcode" InputProps={{
                ...params.InputProps,
                endAdornment: (
                    <React.Fragment>
                        {isLoading ? <CircularProgress color="inherit" size={20}/> : null}
                        {params.InputProps.endAdornment}
                    </React.Fragment>
                ),
            }}
                       error={postcodeError}
                       helperText={postcodeError ? "Not a valid postcode" : ""}
                       onKeyDown={keyPress}/>
        );
    };

    return (
        <Autocomplete
            options={options}
            value={value}
            onChange={onChange}
            onInputChange={onInputChange}
            renderInput={renderInput}
            loading={isLoading}
            filterOptions={x => x}
            freeSolo
            sx={{minWidth: 140}}
            renderOption={(props, option) => {

                return (
                    <li {...props}>
                        <Grid container alignItems="center">
                            <Grid item>
                                <Box
                                    component={LocationOnIcon}
                                    sx={{color: 'text.secondary', mr: 2}}
                                />
                            </Grid>
                            <Grid item xs>
                                {option}
                            </Grid>
                        </Grid>
                    </li>
                );
            }}
        />
    );
}

Let's take a closer look at some of the key features in our code:

Debouncing with Lodash : We've integrated Lodash's debounce function to ensure that we only call Postcodes.io's API once the user has stopped typing for at least 300 milliseconds. This helps to minimize API calls and improve the overall user experience. We then use the data returned from the API to populate the options for our autocomplete dropdown.
Handling validation : The parent component takes care of handling the logic to set the value and error state for validation purposes.
Visual feedback for errors : If the error prop is set to true, we visually indicate that the user has entered an invalid postcode by highlighting the text field in red and displaying an error message.
Autocomplete : We're using Material UI's Autocomplete with the results from Postcodes.io to display a dropdown when the user starts typing a postcode.

Next, we update the Search.tsx component to use our new PostcodeAutocomplete component. Replace the entire component with the following:

export default function Search({initialPostcode, initialRadius}:any) {
    const router = useRouter()
    const [postcode, setPostcode] = useState<string | null>(null);
    const [radius, setRadius] = useState<number>(10);
    const [postcodeError, setPostcodeError] = useState(false);

    useEffect(() => {
        setPostcode(initialPostcode);
        setRadius(initialRadius)
    }, [initialPostcode, initialRadius])

    function handleSubmit() {
        if (!postcode) setPostcodeError(true)
        else isValidPostcode(postcode)
            .then(isValid => {
                setPostcodeError(!isValid);
                if (isValid) router.push(`/pass-rates?postcode=${postcode}&radius=${radius}`)
            });
    }

    return (
        <Paper variant="outlined" sx={{p: 1}}>
            <div style={{display: 'flex', flexFlow: 'row wrap'}}>
                <Box sx={{m: 1, flex: 1}}>
                    <PostcodeAutocomplete postcode={postcode} setPostcode={setPostcode} postcodeError={postcodeError}
                                          setPostcodeError={setPostcodeError} submitForm={handleSubmit}/>
                </Box>
                <FormControl sx={{m: 1, minWidth: 120, flex: 1}}>
                    <InputLabel id="demo-simple-select-label">Radius</InputLabel>
                    <Select
                        labelId="demo-simple-select-label"
                        id="demo-simple-select"
                        value={radius}
                        label="Age"
                        onChange={e => setRadius(e.target.value as number)}
                    >
                        <MenuItem value={5}>5 miles</MenuItem>
                        <MenuItem value={10}>10 miles</MenuItem>
                        <MenuItem value={20}>20 miles</MenuItem>
                        <MenuItem value={30}>30 miles</MenuItem>
                    </Select>
                </FormControl>

                <Box sx={{m: 1, mt: 1.8, flex: 1, display: 'flex', flexDirection: 'column'}}>
                    <Button sx={{alignSelf: 'center'}} variant="outlined" onClick={handleSubmit} size="large">Search</Button>
                </Box>

            </div>

        </Paper>
    );
}

What we've achieved with this change:

Handling Postcode Validation : To improve the accuracy of our user inputs, we've added a new state variable called postcodeError which tracks whether the entered postcode is valid or not. We pass this variable to the PostcodeAutocomplete component to provide visual feedback to the user.
Validating Postcodes with PostcodesAPI : We now validate the postcode in the handleSubmit() method using the isValidPostcode() method in PostcodesAPI. This allows us to ensure that the user has entered a valid UK postcode before submitting the form.

Now, let's test our changes!

Go to http://localhost:3000 and try entering a postcode into the landing page. You should see something like this:

Clicking on one of the postcodes will autofill the postcode for you.

Step 4: Add Search Functionality to the Pass Rates Page

A nice extra feature would be to allow the user to search for a new postcode from the /pass-rates page without having to go back to the landing page.

As we've got the Search and PostcodeAutocomplete functionality in their own components we can easily add this to the /pass-rates page.

Replace the return block in pass-rates.tsx with the following:

return (
    <>
        <Head>
            <meta charSet="utf-8"/>
            <title>Best Driving Test Pass Rates Near Me - Pass Rates</title>
            <meta name="description" content={`Shows the latest pass rates for driving test centres near ${postcode} within a ${radius} mile radius`}/>
        </Head>
        <Box sx={{display: 'flex', justifyContent: 'center', mt: 1, mb: 1}}>
            <Search initialPostcode={postcode} initialRadius={radius}/>
        </Box>

        <Typography variant="h6">Test Centres within {radius} mile radius of {postcode}</Typography>

        <ResultsTable results={results}/>
    </>
);

The only thing we've changed here is that we've added our Search component.

The only difference between our use of Search here and in the landing page is that here we initially populate the postcode and radius fields with the previous search's parameters.

The /pass-rates page will now contain a search form at the top.

Display Nearest Test Centres for Each City

One of the key requirements for our project is to allow users to select the best test centres near a city. We want to do this for two reasons:

If the user wants to look for test centres around a specific city, they don't need to find a postcode for that city.
We can create static pages for each city, which will improve SEO (discussed in a later section) and help increase our chances of appearing at the top of Google's search results.

Step 1: Calculating the Nearest Test Centres for Each City

To calculate the nearest test centres to each city, we need to know the latitude and longitude of each city. We can start by compiling a list of all the cities in the UK from Wikipedia.

Next, we can use Google Maps APIs to search for the latitude and longitude of each city. For instance, to obtain the location data for Birmingham, we send the following request:

https://maps.googleapis.com/maps/api/place/findplacefromtext/json?input=Birmingham&inputtype=textquery&key=<YOUR API KEY>&locationbias=ipbias&fields=name,place_id

The response contains the name of the city and its place ID:

{
  "candidates": [
    {
      "name": "Birmingham",
      "place_id": "ChIJc3FBGy2UcEgRmHnurvD-gco"
    }
  ],
  "status": "OK"
}

Using the place ID, we can then send a second request to retrieve the latitude and longitude of the city:

https://maps.googleapis.com/maps/api/place/details/json?place_id=ChIJc3FBGy2UcEgRmHnurvD-gco&key=<YOUR API KEY>&fields=geometry

The response contains the location data for the city:

{
  "html_attributions": [],
  "result": {
    "geometry": {
      "location": {
        "lat": 52.48624299999999,
        "lng": -1.890401
      },
      "viewport": {
        "northeast": {
          "lat": 52.60869933491674,
          "lng": -1.709829372653529
        },
        "southwest": {
          "lat": 52.38599896742283,
          "lng": -2.017433632448159
        }
      }
    }
  },
  "status": "OK"
}

To calculate which test centres are near each city we first need to decide on what distance we consider "close". I considered anything within a 10-mile radius close.

For each city we need to go through each test centre and calculate the distance between the city and the test centre. If the distance is less than 10 miles then we add that test centre to the list of close test centres.

As before, once you have the data for each city you can use your preferred scripting language to extract the list of nearest test centres for each city.

You'll want to save each city with its closest test centres in individual JSON files. Each JSON file should be the name of the city, in a directory called cities.

E.g. you'll have a file called birmingham.json that looks like this:

{
  "name": "Birmingham",
  "postcode": "B4 7DL",
  "testCentres": [
    {
      "name": "Birmingham (Garretts Green)",
      "passRate": "43.6",
      "distance": 4.767003603076919,
      "mapDetails": {
        "placeId": "ChIJldK_AICwcEgR2-SLap5P9Nw",
        "address": "Granby Ave, Garrett's Green, Birmingham B33 0SJ, United Kingdom",
        "name": "Garretts Green Test Centre",
        "lat": 52.4759864,
        "lng": -1.7783739,
        "rating": 3.7,
        "url": "https://maps.google.com/?cid=15921438124472526043",
        "userRatingsTotal": 58
      }
    },
    {
      "name": "Birmingham (Kings Heath)",
      "passRate": "42.1",
      "distance": 5.493458409608323,
      "mapDetails": {
        "placeId": "ChIJD1VCmci-cEgR77E87hMKPNg",
        "address": "955 Alcester Rd S, Birmingham B14 5JA, United Kingdom",
        "name": "Birmingham Kings Heath Driving Test Centre",
        "lat": 52.4067573,
        "lng": -1.8873253,
        "rating": 3.6,
        "url": "https://maps.google.com/?cid=15581339891512685039",
        "userRatingsTotal": 81
      }
    },
    {
      "name": "Birmingham (Kingstanding)",
      "passRate": "36.7",
      "distance": 4.000310419795066,
      "mapDetails": {
        "placeId": "ChIJHbuU-WOjcEgRtqaHZnumMto",
        "address": "205 Birdbrook Rd, Birmingham B44 9UL, United Kingdom",
        "name": "Birmingham Kingstanding Driving Test Centre",
        "lat": 52.5441402,
        "lng": -1.890445,
        "rating": 3.9,
        "url": "https://maps.google.com/?cid=15722812298035177142",
        "userRatingsTotal": 116
      }
    },
    {
      "name": "Birmingham (Shirley)",
      "passRate": "52.2",
      "distance": 6.287245676525351,
      "mapDetails": {
        "placeId": "ChIJWaNhJae5cEgRb7RLuTDHiaE",
        "address": "374 Stratford Rd, Shirley, Solihull B90 4AQ, United Kingdom",
        "name": "Shirley Driving Test Centre",
        "lat": 52.4051825,
        "lng": -1.8225654,
        "rating": 3.7,
        "url": "https://maps.google.com/?cid=11640053723996861551",
        "userRatingsTotal": 44
      }
    },
    {
      "name": "Birmingham (South Yardley)",
      "passRate": "36.7",
      "distance": 4.019107914150748,
      "mapDetails": {
        "placeId": "ChIJHUTeU0e6cEgRM9dvMwTt1sw",
        "address": "Driving Test Centre, Clay Ln, Birmingham B26 1EA, UK",
        "name": "Driving Test Centre",
        "lat": 52.45498449999999,
        "lng": -1.8098702,
        "rating": 0.0,
        "url": "https://maps.google.com/?q=Driving+Test+Centre&ftid=0x4870ba4753de441d:0xccd6ed04336fd733",
        "userRatingsTotal": 0
      }
    },
    {
      "name": "Birmingham (Sutton Coldfield)",
      "passRate": "36.5",
      "distance": 5.679981743492782,
      "mapDetails": {
        "placeId": "ChIJ7cilVwKlcEgRZ64gPe1-Icw",
        "address": "31-33 Birmingham Rd, Sutton Coldfield B72 1QE, United Kingdom",
        "name": "DVSA Theory Test Centre",
        "lat": 52.558625,
        "lng": -1.826348,
        "rating": 3.0,
        "url": "https://maps.google.com/?cid=14709177415366651495",
        "userRatingsTotal": 23
      }
    },
    {
      "name": "Birmingham (Wyndley)",
      "passRate": "54.5",
      "distance": 4.687979677729207,
      "mapDetails": {
        "placeId": "ChIJZQwbgeSkcEgR6650dA7bWpo",
        "address": "110-116 Boldmere Rd, Boldmere, Sutton Coldfield B73 5UB, UK",
        "name": "110-116 Boldmere Rd",
        "lat": 52.5471217,
        "lng": -1.8411731,
        "rating": 0.0,
        "url": "https://maps.google.com/?q=110-116+Boldmere+Rd,+Boldmere,+The+Royal+Town+of+Sutton+Coldfield,+Sutton+Coldfield+B73+5UB,+UK&ftid=0x4870a4e4811b0c65:0x9a5adb0e7474aeeb",
        "userRatingsTotal": 0
      }
    },
    {
      "name": "Wednesbury",
      "passRate": "39.6",
      "distance": 7.032517930498278,
      "mapDetails": {
        "placeId": "ChIJm07r2l-YcEgRVY5l3eqVSg8",
        "address": "Knowles St, Wednesbury WS10 9HN, United Kingdom",
        "name": "DVSA Driving Test Centre",
        "lat": 52.5555689,
        "lng": -2.0128767,
        "rating": 2.8,
        "url": "https://maps.google.com/?cid=1101857894814813781",
        "userRatingsTotal": 85
      }
    }
  ]
}

and another called aberdeen.json that looks like this:

{
  "name": "Aberdeen",
  "postcode": "AB24 5BA",
  "testCentres": [
    {
      "name": "Aberdeen North",
      "passRate": "57.3",
      "distance": 2.4724386179137534,
      "mapDetails": {
        "placeId": "ChIJC3xww_sNhEgR7iXmPVZSNVA",
        "address": "8GT, Balgownie Rd, Bridge of Don, Aberdeen, United Kingdom",
        "name": "DVSA",
        "lat": 57.1856443,
        "lng": -2.0964023,
        "rating": 4.1,
        "url": "https://maps.google.com/?cid=5779616227159057902",
        "userRatingsTotal": 7
      }
    },
    {
      "name": "Aberdeen South (Cove)",
      "passRate": "64.5",
      "distance": 4.274066387101663,
      "mapDetails": {
        "placeId": "ChIJTY7RiP4PhEgRDr0PT-qctJE",
        "address": "Moss Rd, Aberdeen AB12 3GQ, United Kingdom",
        "name": "DVSA Driving Test Centre",
        "lat": 57.0884979,
        "lng": -2.1077442,
        "rating": 4.3,
        "url": "https://maps.google.com/?cid=10499189161470180622",
        "userRatingsTotal": 19
      }
    }
  ]
}

Step 2: Create a Page To Show All Cities

The first thing we'll do is create a page that lists all cities. Later we'll use this page to navigate the user to each corresponding city's results page.

Before we create the page we'll create the necessary lib function to get all the city names. We do this in a similar way to how we got all the test centre data.

Create a file called cities.ts in the lib directory. This should contain the following:

import fs from 'fs';
import path from 'path';

const citiesDirectory = path.join(process.cwd(), 'cities');

export function getAllCityIds() {
    const fileNames = fs.readdirSync(citiesDirectory);

    return fileNames.map((fileName) => {
        return fileName.replace(/\.json$/, '');
    });
}

Here we exposed a function called getAllCityIds, which returns the name of all cities, with each city name taken from the name of the JSON files that we saved in the cities directory in the previous step.

We can then create a page to show all the cities. Create a new file, cities.tsx under the pages directory. This should contain the following:

import {Box, Typography} from "@mui/material";
import {getAllCityIds} from "../lib/cities";
import Head from "next/head";
import React from "react";

export async function getStaticProps() {
    const cityIds = getAllCityIds();
    return {
        props: {
            cityIds,
        },
    };
}

interface CitiesProps {
    cityIds: string[]
}

function capitalizeFirstLetter(string: string) {
    return string.split(" ").map(word => word.charAt(0).toUpperCase() + word.slice(1)).join(" ")
}

export default function cities({cityIds}: CitiesProps) {
    return (
        <>
            <Head>
                <meta charSet="utf-8"/>
                <title>Best Driving Test Pass Rates Near Me - Pass Rates</title>
                <meta name="description"
                      content={`A list of all the cities in the UK, each linking to the latest pass rates for driving centres in that area.`}/>
            </Head>
            <Box sx={{display: 'flex', flexDirection: 'column'}}>
                <Typography variant="h2">Cities</Typography>
                <ul>
                    {cityIds.map(id => <li key={id}>{capitalizeFirstLetter(id)}</li>)}
                </ul>
            </Box>
        </>
    )
}

Here we load the city names using getStaticProps (so on build time), and render a list containing each city name.

Navigate to http://localhost:3000/cities. You will see a page containing all the city names.

Step 3: Create a Page For Each City

Now that we have a list of all the cities, we need to create a page for each city.

To achieve this, we will create a static page for each city that we have JSON data for in our cities directory. These pages will have the URL format /pass-rates/<city name>.

This means that Google can easily index these pages and return them when someone searches for the best driving test centres near a city.

You might have seen that other websites do something similar to this, where they create static pages for commonly searched criteria to increase their chances of appearing at the top of Google's search results (amongst other benefits).

One example of this is crontab.guru which has static pages for commonly used cron expressions, e.g. every 5 minutes, which can be found at https://crontab.guru/every-5-minutes.

If you search in Google for cron job every 5 minutes, then that page will be the first result.

Thankfully we don't have to create each city's page by hand. We can easily accomplish this by using Next.js's getStaticPaths function, which allows us to specify a list of paths that should be generated as pages during build time.

To start, we will need to generate a list of city names. In the cities.ts file located in the lib directory, add the following function:

export function getAllCityPaths() {
  const fileNames = fs.readdirSync(citiesDirectory);

  return fileNames.map((fileName) => {
    return {
      params: {
        id: fileName.replace(/\.json$/, ''),
      },
    };
  });
}

This function reads the contents of the cities directory and returns an array of city names by removing the .json extension from the file names. To use getStaticPaths we need to have each city in the format above, which is each city has an object containing a params property, which contains an id property, which is set to the name of the page.

Next, we need to create a new page that will display the test centres for a particular city. Create a directory called pass-rates under the pages directory, and then create a file called [id].tsx under the pass-rates directory. It is important that the square brackets are included in the name, as that tells Next.js that we want the page name to be derived from the list passed to getStaticPaths.

The [id].tsx file should contain the following:

import {getAllCityPaths} from "../../lib/cities";
import React from "react";

export default function City({id}: any) {
  return <>
    {id}
  </>;
}

export async function getStaticPaths() {
  const paths = getAllCityPaths();
  return {
    paths,
    fallback: false,
  };
}

export async function getStaticProps({params}: any) {
  const id = params.id;
  return {
    props: {
      id
    }
  }
}

This file exports two functions, getStaticPaths and getStaticProps. getStaticPaths is used to generate the paths for each city, and getStaticProps is used to fetch the data for each city based on the path. For now, we are just rendering the name of the city.

As we placed our [id].tsx file under the pass-rates directory, any generated page will be found at /pass-rates/<page>.

Let's test this out! Navigating to the generated page for Birmingham. Go to http://localhost:3000/pass-rates/birmingham. You should see a page showing the city name.

Show the Nearest Test Centres For Each City

Now that we have the generated pages, let's populate each of them with the nearest test centre data.

First, we'll need to add another function to cities.ts in the lib directory to get the data for each city. Add the following function:

export function getCityData(id: string) {
  const fullPath = path.join(citiesDirectory, `${id}.json`);
  const fileContents = JSON.parse(fs.readFileSync(fullPath, 'utf8'));
  // Combine the data with the id
  return {
    id,
    ...fileContents,
  };
}

This function parses the JSON from each city file and returns it, with the city name set as the id property.

Next, let's change the getStaticProps function to call getCityData, and then change the React component to render the results. Replace [id].tsx completely with the following:

import {getAllCityPaths, getCityData} from "../../lib/cities";
import React from "react";
import Head from "next/head";
import {Box} from "@mui/system";
import ResultsTable from "../../components/ResultsTable";
import Search from "../../components/Search";
import {Typography} from "@mui/material";

export default function City({cityData}:any) {
  return <>
    <Head>
      <meta charSet="utf-8"/>
      <title>Best Driving Test Pass Rates Near {cityData.name}</title>
      <meta name="description" content={`Shows the latest pass rates for driving test centres near ${cityData.name} within a 10 mile radius`}/>
      <link rel="canonical" href={`https://drivingpassrate.co.uk/pass-rates/${cityData.name}`}/>
    </Head>
    <Box sx={{display: 'flex', justifyContent: 'center', mt: 1, mb: 1}}>
      <Search initialPostcode={cityData?.postcode} initialRadius={10}/>
    </Box>

    <Typography variant="h6" >Test Centres within 10 mile radius of {cityData.name}</Typography>

    <ResultsTable results={cityData.testCentres}/>
  </>;
}

export async function getStaticPaths() {
  const paths = getAllCityPaths();
  return {
    paths,
    fallback: false,
  };
}

export async function getStaticProps({ params }:any) {
  const cityData = getCityData(params.id);
  return {
    props: {
      cityData
    }
  }
}

What we've changed:

In getStaticProps, we are now calling getCityData to retrieve the test centre data for the selected city. We then add this data to the props.
We are rendering the Search component, which is used to search for a test centre by postcode. We are passing in the postcode for the selected city so that it is used as the default search term.
We are passing the test centre data to the ResultsTable component, which is responsible for rendering the table that shows the pass rates for each test centre. We are only passing the test centres that are nearest to the selected city.

Going to http://localhost:3000/pass-rates/birmingham will now show a page very similar to the /pass-rates page:

Step 4: Link Each City Name in /cities

The next step is to go back to the /cities page and add a button for each city that links to the city's individual page.

Open the cities.tsx file under pages and replace the React component with the following:

export default function cities({cityIds}: CitiesProps) {
  return (
          <>
            <Head>
              <meta charSet="utf-8"/>
              <title>Best Driving Test Pass Rates Near Me - Pass Rates</title>
              <meta name="description"
                    content={`A list of all the cities in the UK, each linking to the latest pass rates for driving centres in that area.`}/>
            </Head>
            <Box sx={{display: 'flex', flexDirection: 'column'}}>
              <Typography variant="h2">Cities</Typography>
              <Box>
                {cityIds.map(id => {
                  return (
                          <Link href={`/pass-rates/${id}`} passHref key={id} style={{textDecoration: 'none', color: 'inherit'}}>
                            <ListItemButton >
                              <ListItemText primary={capitalizeFirstLetter(id)}/>
                            </ListItemButton>
                          </Link>)
                })
                }
              </Box>
            </Box>
          </>
  )
}

This code changes each city name into a button that links to the corresponding city page.

Navigate to http://localhost:3000/cities. You will see the updated page with a button for each city.

Clicking on any button will now bring you to that city's page.

You might notice that it's slow to render the city pages. This is because Next.js is rendering the pages on the fly at runtime when running the app in dev mode. When the application is deployed in production mode it will be much faster as each page will be statically rendered at build time.

Step 5: Add a Link to /cities on the Landing Page

Finally, let's add a link to our /cities page from the landing page.

Open index.tsx and add the following code under the <Search/> element:

<Box sx={{m: 2, mt: 1.8, display: 'flex', flexDirection: 'column'}}>
  <Link href={`/cities`} passHref>
    <Button sx={{alignSelf: 'center'}} variant="outlined" component="a">See all cities</Button>
  </Link>
</Box>

Visit http://localhost:3000 to see the updated landing page with the "See all cities" button:

Conclusion

This completes part 2!

Let's review what we've done here:

Added the ability to search by postcode and radius on the landing page and the pass rates page
Added autocomplete to the postcodes search field
Created a page for each city, showing the nearest test centres
Created a page listing all cities, and add a link to this page from the landing page

You can find all the code used in this post in my repo here. As well, you can find a live version of the application we built here.

Look out for part 3, where we'll be adding the finishing touches and deploying our application!

Updated: Part 3 is now available. Read it here.

Till next time!

Creating a Project Using Public Data for Fun and Profit: Part 1

Shane Jennings — Fri, 14 Apr 2023 15:08:16 +0000

In this post, we'll explore how to create an engaging website using public data that you can monetize and use to impress potential employers. You'll learn the benefits of developing a personal project using public data and follow an example of building an app from scratch to deployment.

Throughout this blog, we'll walk through a step-by-step guide on how to build a website that allows users to search for driving test centres near them and show which centres have the best pass rates. We'll be using the UK Government's public data on driving test centre pass rates. I built this app because I'm learning to drive, and I wanted to know which test centre would give me the best chance of passing.

This will be a big post, so I've split it into 4 parts. The other parts will be coming soon!

In this part, we will cover:

Setting up the initial project
Retrieving and formatting the public test centre data
Creating and styling the landing page

All the code referenced in this post can be found in my repo here. As well, you can find a live version of the application we build here.

Why Should I Create My Own Personal Project?

Developers create personal projects for several reasons, including showcasing their skills to potential employers, learning new technologies, having fun, and making a profit. Often, a project can fulfil all these reasons simultaneously. Building a personal project is one of the best ways to create a portfolio that can help you break into the software industry and land a job.

Why Should I Use Public Data?

If you're looking for project ideas, public data can be an excellent source of inspiration. Public data is easy to obtain, it's free, and when combined with other data, it can provide substantial value to users. Even presenting data in a user-friendly format can be a significant win, considering that public data is often not presented this way.

Some examples of sources of public data are government and video games. For example, you can find health data, such as the number of vaccinations in England, on the UK Government's Coronavirus Dashboard here. Another example is tax data, such as council tax data, which can be found here. Alternatively, you can find video game data, such as the World of Warcraft API or the PokAPI for Pokmon.

You could also scrape data from websites if they don't offer a nice way to get their data, depending on their data usage policies, however, that deserves its own separate post. If you're interested in web scraping, you can find a tutorial here from FreeCodeCamp.

There are many other sources of public data, and the key is to choose data that interests you or solves a particular problem you're facing. That way you're much more likely to stick with the project!

The Requirements

Before we begin, let's set out the requirements:

The user can see the best driving test centres around them, based on their postcode.
The user can see the best driving test centres around popular cities.
The user can see the Google Reviews rating for each test centre.
The user can easily see where the test centre is located on Google Maps.

Set Up

For this application, we'll use Next.js as the framework and Material UI for the design. If you haven't used Next.js before, don't worry. Most of it will be clear if you've used React before.

I chose Next.js for its static site generation, which makes it lightning-fast. Since most of the data we use here is not dynamic, we can generate our pages at build time. This not only makes the application load extremely fast, but it's also good for SEO (which we'll cover later), which is crucial if we want to monetize our app later. It's also easy to host, as we'll see later.

Throughout this post, we'll explore how we can use Next.js for static site generation. However, you can read more about Next.js here.

To start, let's initialise our project:

> npx create-next-app@latest --ts

Once completed, we can start up our application by running:

npm run dev

Opening localhost:3000 should then show us the Next.js example page:

Your directory structure should look like this:

Getting the Public Data

Step 1: Acquiring Test Centre Data

Please note that in this section I'll be deliberately brief on the details. This is partly because the point of this post isn't to copy what I did entirely, but rather to provide inspiration, but also because I don't want this post to be any longer than it needs to be. I leave the massaging of data as an exercise for the reader.

To build our web app, we first need to obtain data on the driving test centres. This data can be downloaded from the UK government's page on driving test pass rates, which can be found here. Specifically, we need the "Car pass rates by gender, month and test centre" document which provides the total pass rate for men and women for each driving test centre per year.

This document contains a tab for each year, with a record of the total pass rate for all men and women that year for each driving test centre.

As an example, we can see that Aberdeen North had a total pass rate of 48.2% for the year 2022-23.

Once you have this data, you can use your preferred scripting language to extract a list of each test centre name with its corresponding total pass rate.

Step 2: Retrieving Test Centre Location Data

In order to calculate the distance between the user and each test centre, we need to determine the location of each test centre. One way to accomplish this is by using the Google Maps Places API. While the API is not free, it offers 28,500 free map-loads per month, which should be more than sufficient for our needs.

To use the Places API, you need to obtain your own API key, which you can do by following the instructions provided here.

We'll be using two endpoints of the Places API - findplacefromtext and details. These endpoints return the following fields:

geometry: The longitude and latitude of the test centre location.
rating: The Google reviews rating for the location.
user_ratings_total: The total number of reviews left for the location.
url: The link to the location in Google Maps.

We can use the findplacefromtext endpoint to search for our test centres by name and retrieve a list of candidates with their corresponding name and place ID. For example, when searching for the Mill Hill (London) driving test centre, you can query the following URL:

https://maps.googleapis.com/maps/api/place/findplacefromtext/json?input=Mill%20Hill%20(London)%20Driving%20Test%20Centre&inputtype=textquery&key=<YOUR API KEY>&locationbias=ipbias&fields=name,place_id

This query returns:

{
  "candidates": [
    {
      "name": "Mill Hill Driving Test Centre",
      "place_id": "ChIJXdhrutoWdkgRh_OPEmlGwmc"
    }
  ],
  "status": "OK"
}

Next, we can use the place ID to retrieve all the details about that test centre using the details endpoint. For example, you can query the following URL to retrieve details for the Mill Hill (London) driving test centre:

https://maps.googleapis.com/maps/api/place/details/json?place_id=ChIJXdhrutoWdkgRh_OPEmlGwmc&key=<YOUR API KEY>&fields=geometry,rating,url,user_ratings_total

This query returns:

{
  "html_attributions": [],
  "result": {
    "geometry": {
      "location": {
        "lat": 51.6103168,
        "lng": -0.2469914999999999
      },
      "viewport": {
        "northeast": {
          "lat": 51.6115767802915,
          "lng": -0.2457076697084979
        },
        "southwest": {
          "lat": 51.6088788197085,
          "lng": -0.248405630291502
        }
      }
    },
    "rating": 4.7,
    "url": "https://maps.google.com/?cid=7476615748485378951",
    "user_ratings_total": 350
  },
  "status": "OK"
}

For test centres where I got more than one candidate I hand-picked the right one - however, you could write your own algorithm to programmatically figure this out.

Step 3: Combining the Data

We need to combine the data we collected earlier to create a list of all the test centres. We'll include their name, pass rate, lat long, rating, number of ratings, and the Google Maps URL in a JSON file. Let's save this file in a directory called testcentres in the root directory.

Here's an example of what the JSON file should look like:

[
  {
    "name": "Aberdeen North",
    "passRate": "57.3",
    "mapDetails": {
      "lat": 57.1856443,
      "lng": -2.0964023,
      "rating": 4.1,
      "url": "https://maps.google.com/?cid=5779616227159057902",
      "userRatingsTotal": 7
    }
  },
  {
    "name": "Aberdeen South (Cove)",
    "passRate": "64.5",
    "mapDetails": {
      "lat": 57.0884979,
      "lng": -2.1077442,
      "rating": 4.3,
      "url": "https://maps.google.com/?cid=10499189161470180622",
      "userRatingsTotal": 19
    }
  },
  ...
]

Creating the Landing Page

In this section, we'll start creating our landing page.

Step 1: Install the Necessary NPM Packages

We will need to install both Material UI and Emotion:

> npm install @mui/material
> npm install @mui/icons-material
> npm install @emotion/react
> npm install @emotion/styled

Step 2: Style the Landing Page

Next, we'll update our _app.tsx file and index.tsx file to use Material UI components and styles. The changes to _app.tsx will apply our Material UI theme and our Flexbox styling to all pages.

Replace _app.tsx with the following:

import '../styles/globals.css'
import type {AppProps} from 'next/app'
import {Container, createTheme, ThemeProvider} from "@mui/material";
import Head from "next/head";
import Box from "@mui/material/Box";

const theme = createTheme({
    typography: {
        fontFamily: [
            "-apple-system", "BlinkMacSystemFont", "Segoe UI", "Roboto", "Oxygen", "Ubuntu", "Cantarell", "Fira Sans", "Droid Sans", "Helvetica Neue", "sans-serif"
        ].join(",")
    },
    palette: {
        primary: {
            light: '#6985ff',
            main: '#008000',
            dark: '#0031c3',
            contrastText: '#ffffff',
        },
        secondary: {
            light: '#9fcfff',
            main: '#689eff',
            dark: '#2770cb',
            contrastText: '#ffffff',
        },
    },
});

function MyApp({Component, pageProps}: AppProps) {
    return <>
        <Head>
            <title>Best Driving Test Pass Rates Near Me</title>
        </Head>
        <ThemeProvider theme={theme}>
            <Box sx={{display: 'flex', flexDirection: 'column', minHeight: '100vh'}}>
                <Container fixed sx={{display: 'flex', flex: 1, flexDirection: 'column'}}>
                    <Component {...pageProps} />
                </Container>
            </Box>
        </ThemeProvider>
    </>
}

export default MyApp

Replace index.tsx with the following:

import type {NextPage} from 'next'
import styles from '../styles/Home.module.css'
import {Typography} from "@mui/material";

const Home: NextPage = () => {

  return (
          <div className={styles.content}>
            <div style={{display: 'flex'}}>
              <Typography variant="h2" sx={{m: 1, fontWeight: 'bold'}}>Find The Best <span
                      style={{color: 'green'}}>Pass Rates</span> Near You <br/>🚗</Typography>
            </div>
          </div>
  )
}

export default Home

We then need to change the Home.modules.css file. This file will contain specific styles for our landing page. Replace it with the following:

.container {
  padding: 0 2rem;
}

.app {
  display: flex;
  flex: 1;
  text-align: center;
}

.content {
  text-align: center;
  display: flex;
  flex: 1;
  flex-direction: column;
  align-items: center;
  justify-content: center;
  font-size: calc(10px + 2vmin);
}

.main {
  min-height: 100vh;
  padding: 4rem 0;
  flex: 1;
  display: flex;
  flex-direction: column;
  justify-content: center;
  align-items: center;
}

Finally, we need to change the globals.css file. This file will contain global styles that will be applied to all pages. Replace it with the following:

html,
body {
  padding: 0;
  margin: 0;
  font-family: -apple-system, BlinkMacSystemFont, Segoe UI, Roboto, Oxygen,
  Ubuntu, Cantarell, Fira Sans, Droid Sans, Helvetica Neue, sans-serif;
}

* {
  box-sizing: border-box;
}

You can also delete the api folder and hello.ts file inside of it, as we will not be creating an API for this application.

Run npm run dev and navigate to localhost:3000. You will now see our styled landing page.

Showing the Nearest Test Centres for a Specific Postcode

One of our main requirements is to enable users to view the best driving test centres in their area, based on their postcode. To achieve this, we need to find the nearest test centres based on a given postcode and display the results on a dedicated web page.

Step 1: Create a Function To Get Location Data By Postcode

To do this, we can use postcodes.io, an open-source postcode and geolocation API for the UK, to find the nearest test centres based on a postcode. Postcodes.io API provides us with approximate longitude and latitude data based on a postcode, and vice versa. It even offers autocomplete data for partial postcodes, which we'll use later.

To start, we'll define a file that contains all the logic to connect to postcodes.io API and gets the location data for a specific postcode.

We'll create a new directory called api and a new file called PostcodesAPI.ts inside it, which should contain the following:

import LongLat from "./LongLat";

const baseUrl = 'https://api.postcodes.io/postcodes/';

export async function getLongLatFromPostcode(postcode: string) {
  const response = await getPostcodeResponse(postcode);
  if (!response) return;
  return response.result;
}

const getPostcodeResponse = async (postcode: string): Promise<PostcodeResponseDTO | null> => {
  return await fetch(`${baseUrl}${postcode}`, {
    headers: {
      'Content-Type': 'application/json',
      'Accept': 'application/json'
    }
  }).then(response => {
    return response.json();
  }).catch(error => {
    console.error(error);
    return null;
  })
}

interface PostcodeResponseDTO {
    result: LongLat
}

Next, we'll create a new directory called lib and add a file called LongLat.ts. This file contains the following code:

export default interface LongLat {
    longitude: number
    latitude: number
}

Step 2: Create a Function To Get Test Centre Data

We need to create a function that returns the list of objects from the testcentres.json file that we generated earlier.

Create a new file inside the lib directory called testcentres.ts containing the following:

import path from "path";
import fs from "fs";
import {TestCentre} from "./TestCentre";

const directory = path.join(process.cwd(), 'testcentres');

export function getTestCentres() {
    const fullPath = path.join(directory, `testcentres.json`);
    const testCentres : TestCentre[] = JSON.parse(fs.readFileSync(fullPath, 'utf8'));
    return testCentres;
}

We also need to create another file called TestCentre.ts, which contains the following:

interface MapDetails {
    lat: number
    lng: number
    rating: number
    url: string
    userRatingsTotal: number
}

export interface TestCentre {
    name: string
    passRate: string
    mapDetails: MapDetails
}

export interface TestCentreWithDistance extends TestCentre {
    distance: number
}

Step 3: Create A Page to Show Results

Now, we can create a new page called pass-rates.tsx in the pages directory. This will display the details of all test centres nearest to the supplied postcode, within the supplied radius.

The pass-rates.tsx file should contain the following code:

import React, {useEffect} from "react";
import {useRouter} from 'next/router'
import Head from "next/head";
import {Typography} from "@mui/material";

import {getLongLatFromPostcode} from "../api/PostcodesAPI";
import LongLat from "../lib/LongLat";
import {getTestCentres} from "../lib/testcentres";
import {TestCentre, TestCentreWithDistance} from "../lib/TestCentre";

export async function getStaticProps() {
    const testCentres = getTestCentres();
    return {
        props: {
            testCentres
        },
    };
}

function getTestCentresWithinRadius(usersLongLat: LongLat, radius: number, testCentres: TestCentre[]) : TestCentreWithDistance[] {
    return testCentres.flatMap(centre => {
        const distance = getDistanceFromLatLon(usersLongLat.latitude, usersLongLat.longitude, centre.mapDetails.lat, centre.mapDetails.lng);
        if(distance > radius) return [];
        const centreWithDistance = structuredClone(centre) as TestCentreWithDistance;
        centreWithDistance.distance = distance;
        return centreWithDistance;
    })
}

function getDistanceFromLatLon(lat1: number, lon1: number, lat2: number, lon2: number): number {
    const R = 6371; // Radius of the earth in km
    const dLat = deg2rad(lat2 - lat1); // deg2rad below
    const dLon = deg2rad(lon2 - lon1);
    const a =
        Math.sin(dLat / 2) * Math.sin(dLat / 2) +
        Math.cos(deg2rad(lat1)) * Math.cos(deg2rad(lat2)) *
        Math.sin(dLon / 2) * Math.sin(dLon / 2);
    const c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1 - a));
    return (R * c) * 0.62137119; // Convert distance from km to miles
}

function deg2rad(deg: number): number {
    return deg * (Math.PI / 180);
}

export default function Results({testCentres}:any) {
    const router = useRouter()
    const {postcode, radius} = router.query;
    const [results, setResults] = React.useState<TestCentreWithDistance[]>([]);

    const getResults = async () => {
        if (!postcode) return;
        if (!radius) return;
        const usersLongLat = await getLongLatFromPostcode(postcode as string);
        if (!usersLongLat) return;

        setResults(getTestCentresWithinRadius(usersLongLat, parseInt(radius as string), testCentres));
    }

    useEffect(() => {
        getResults();
    }, [postcode, radius])

    return (
        <>
            <Head>
                <meta charSet="utf-8"/>
                <title>Best Driving Test Pass Rates Near Me - Pass Rates</title>
                <meta name="description" content={`Shows the latest pass rates for driving test centres near ${postcode} within a ${radius} mile radius`}/>
            </Head>

            <Typography variant="h6">Test Centres within {radius} mile radius of {postcode}</Typography>
            <ul>

            {
                results.map(
                    r => <li key={r.name}>Name: {r.name}, Distance: {r.distance}, Rating: {r.mapDetails.rating} ({r.mapDetails.userRatingsTotal} reviews), mapUrl: {r.mapDetails.url}</li>
                )
            }
            </ul>

        </>
    );
}

Let's go through in detail what is happening here.

To get the supplied postcode and radius we use Next.js's Router in the line const {postcode, radius} = router.query. This reads the query parameters from the URL and copies them into parameters with the same name.

For example, if the URL is http://localhost:3000/pass-rates?postcode=NG16JX&radius=10, then the postcode variable will be set to NG16JX, and the radius variable will be set to 10.

To get the test centre data we use the function we defined in the previous step, getTestCentres. We use this in combination with Next.js's getStaticProps function. getStaticProps is a special function that is run when the page is statically generated at build time. That means the test centre's data is already loaded before a user loads the page, which drastically helps speed up the rendering time for a page. You can read more about this here.

To know which test centres are within the specified radius we first calculate the latitude and longitude of the specified postcode by calling the getLongLatFromPostcode function we defined earlier. Using the longitude and latitude, along with the radius and all the test centre data, we calculate the distance between the specified postcode and each test centre using the Haversine formula. We then convert that distance into miles (as our radius measurement is miles). We then use this distance to filter out any test centre with a distance greater than the specified radius.

Finally, we display all the results in a list.

We can now navigate to the /pass-rates page and pass in a postcode and a radius. E.g. to find all test centres within a 10-mile radius of Nottingham we can navigate to: http://localhost:3000/pass-rates?postcode=NG16JX&radius=10

Which will show us a page will all the test centres within 10 miles of Nottingham.

Try it out with your own postcode!

Step 4: Showing the Results as a Table

While our /pass-rates page does show the correct results, it's not very pretty. Let's use a table instead to render the data.

To do this, let's add a new file ResultsTable.tsx to our components directory, which should contain the following:

import * as React from 'react';
import Box from '@mui/material/Box';
import Table from '@mui/material/Table';
import TableBody from '@mui/material/TableBody';
import TableCell from '@mui/material/TableCell';
import TableContainer from '@mui/material/TableContainer';
import TableHead from '@mui/material/TableHead';
import TableRow from '@mui/material/TableRow';
import TableSortLabel from '@mui/material/TableSortLabel';
import Paper from '@mui/material/Paper';
import {visuallyHidden} from '@mui/utils';
import {TestCentreWithDistance} from "../lib/TestCentre";
import {Tooltip} from "@mui/material";
import HelpIcon from '@mui/icons-material/HelpOutline';
import MapIcon from '@mui/icons-material/Map';

interface Data {
    name: string
    passRate: number
    rating: string
    distance: number
    mapURL: string
}

function descendingComparator<T>(a: T, b: T, orderBy: keyof T) {
    if (b[orderBy] < a[orderBy]) {
        return -1;
    }
    if (b[orderBy] > a[orderBy]) {
        return 1;
    }
    return 0;
}

type Order = 'asc' | 'desc';

function getComparator<Key extends keyof any>(
    order: Order,
    orderBy: Key,
): (
    a: { [key in Key]: number | string },
    b: { [key in Key]: number | string },
) => number {
    return order === 'desc'
        ? (a, b) => descendingComparator(a, b, orderBy)
        : (a, b) => -descendingComparator(a, b, orderBy);
}

// This method is created for cross-browser compatibility, if you don't
// need to support IE11, you can use Array.prototype.sort() directly
function stableSort<T>(array: readonly T[], comparator: (a: T, b: T) => number) {
    const stabilizedThis = array.map((el, index) => [el, index] as [T, number]);
    stabilizedThis.sort((a, b) => {
        const order = comparator(a[0], b[0]);
        if (order !== 0) {
            return order;
        }
        return a[1] - b[1];
    });
    return stabilizedThis.map((el) => el[0]);
}

interface HeadCell {
    disablePadding: boolean;
    id: keyof Data;
    label: string;
    numeric: boolean;
}

const headCells: readonly HeadCell[] = [
    {
        id: 'name',
        numeric: false,
        disablePadding: true,
        label: 'Test Centre',
    },
    {
        id: 'passRate',
        numeric: true,
        disablePadding: true,
        label: 'Pass Rate',
    },
    {
        id: 'rating',
        numeric: true,
        disablePadding: true,
        label: 'Rating'
    },
    {
        id: 'distance',
        numeric: true,
        disablePadding: true,
        label: 'Distance (miles)',
    }
];

interface EnhancedTableProps {
    onRequestSort: (event: React.MouseEvent<unknown>, property: keyof Data) => void;
    order: Order;
    orderBy: string;
    rowCount: number;
}

function EnhancedTableHead(props: EnhancedTableProps) {
    const {order, orderBy, onRequestSort} =
        props;
    const createSortHandler =
        (property: keyof Data) => (event: React.MouseEvent<unknown>) => {
            onRequestSort(event, property);
        };

    return (
        <TableHead>
            <TableRow>
                {headCells.map((headCell) =>
                    (<TableCell
                            key={headCell.id}
                            align={headCell.numeric ? 'right' : 'left'}
                            sortDirection={orderBy === headCell.id ? order : false}
                        >
                            <TableSortLabel
                                active={orderBy === headCell.id}
                                direction={orderBy === headCell.id ? order : 'asc'}
                                onClick={createSortHandler(headCell.id)}
                            >
                                {headCell.label !== 'Rating' ?
                                    headCell.label : <>{'Rating'} <Tooltip
                                        title="Ratings from Google"><HelpIcon sx={{ml: 1}}/></Tooltip></>}
                                {orderBy === headCell.id ? (
                                    <Box component="span" sx={visuallyHidden}>
                                        {order === 'desc' ? 'sorted descending' : 'sorted ascending'}
                                    </Box>
                                ) : null}
                            </TableSortLabel>
                        </TableCell>
                    ))}
            </TableRow>
        </TableHead>
    );
}

interface ResultsTableProps {
    results: TestCentreWithDistance[]
}

export default function ResultsTable({results}: ResultsTableProps) {
    const [order, setOrder] = React.useState<Order>('desc');
    const [orderBy, setOrderBy] = React.useState<keyof Data>('passRate');

    const r = results.map(result => {
        return {
            name: result.name, passRate: result.passRate,
            rating: result.mapDetails.rating ? `${result.mapDetails.rating} (${result.mapDetails.userRatingsTotal} reviews)` : ``,
            mapURL: result.mapDetails.url,
            distance: result.distance
        };
    })

    const handleRequestSort = (
        event: React.MouseEvent<unknown>,
        property: keyof Data,
    ) => {
        const isAsc = orderBy === property && order === 'asc';
        setOrder(isAsc ? 'desc' : 'asc');
        setOrderBy(property);
    };

    return (
        <>
            <TableContainer component={Paper}>
                <Table
                    size={"medium"}
                    aria-labelledby="tableTitle"
                    style={{ tableLayout: 'auto' }}
                    padding={'normal'}

                >
                    <EnhancedTableHead
                        order={order}
                        orderBy={orderBy}
                        onRequestSort={handleRequestSort}
                        rowCount={results.length}
                    />
                    <TableBody>
                        {/* if you don't need to support IE11, you can replace the `stableSort` call with:
              rows.slice().sort(getComparator(order, orderBy)) */}
                        {stableSort(r, getComparator(order, orderBy))
                            .map((row) => {
                                return (
                                    <TableRow
                                        hover
                                        tabIndex={-1}
                                        key={row.name}
                                    >
                                        <TableCell>{row.name}<MapIcon onClick={() => window.open(row.mapURL)}/></TableCell>
                                        <TableCell align="right">{row.passRate}%</TableCell>
                                        <TableCell align="right">{row.rating}</TableCell>
                                        <TableCell
                                            align="right">{Math.round((row.distance + Number.EPSILON) * 100) / 100}</TableCell>
                                    </TableRow>
                                );
                            })}
                    </TableBody>
                </Table>
            </TableContainer>
        </>
    );
}

There's a lot of code in this file, however, a lot of it is copied from the "Sorting & Selecting" example given for tables in Material UI here.

The most important section to focus on is the tsx returned from the function, showing how our table is rendered and with what cells and rows.

The table contains functionality for the user to sort by each column, and each column contains a map icon, linking the user to the test centre on Google Maps.

Navigating to http://localhost:3000/pass-rates?postcode=NG16JX&radius=10 now shows us a table with the same results as before.

Conclusion

This completes part 1!

Let's review what we've done here:

Set up the initial project
Retrieved and formatted the public test centre data
Created and styled the landing page

You can find all the code used in this post in my repo here. As well, you can find a live version of the application we built here.

Look out for part 2, where we'll be adding search functionality and displaying the nearest test centres for each city!

Updated: Part 2 is now available. Read it here.

Till next time!

How To Deploy A Next.js App Using Spring Boot

Shane Jennings — Tue, 04 Apr 2023 20:45:52 +0000

Next.js is now my favourite way to build front-end applications. With Next.js you can develop a full stack application, including your own API, however, sometimes you want to serve the front end by adding static assets to a backend application.

In this guide, we will learn how to serve Next.js static assets from a Spring Boot application.

How we usually deploy Next.js applications

Before we walk through how to serve Next.js assets from Spring Boot we should first understand how Next.js applications are usually deployed.

When we want to deploy a Next.js application we must first run next build to build the application. As described in the official documentation here, this produces:

HTML files for pages using getStaticProps or Automatic Static Optimization
CSS files for global styles or for individually scoped styles
JavaScript for pre-rendering dynamic content from the Next.js server
JavaScript for interactivity on the client-side through React

We must then run next start. This spins up a Node.js server that handles everything for us, including serving the static assets, handling API endpoints, routing, etc.

This is the recommended way to deploy Next.js applications however, in some cases, we don't want to have a separate server for our front end if we're already implementing back-end functionality in a different place, especially if we're not using Node.js.

In the next section, we'll explore how to serve our Next.js application using Spring Boot, without requiring a Node.js server.

Deploying a Next.js application using Spring Boot

The easiest way to explain how to do this is through a demo. In this section, I'll walk through a step-by-step guide.

Setting up

First, let's create a folder called spring-boot-nextjs-demo:

mkdir spring-boot-nextjs-demo
cd spring-boot-nextjs-demo

Then we'll create a Spring Boot application using the Spring Initializr tool.

You can create the project with all of the required tools using the link here. Click "GENERATE" and extract the zip to the spring-boot-nextjs-demo folder.

Next, let's create the Next.js app. Make sure you're in the spring-boot-nextjs-demo folder and run:

npx create-next-app@latest --ts

Call the app nextjs-app and follow the rest of the defaults.

Setting up the Maven modules

In this project we will have the following Maven structure:

a parent Maven project called spring-boot-nextjs-demo
a child module called nextjs-app, which is responsible for building the Next.js application
a child module called spring-boot-app, which is responsible for building our Spring Boot application and adding our Next.js assets to the jar.

Parent Maven setup

We'll create a pom.xml file in the root folder, spring-boot-nextjs-demo, which contains the following:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.0.0</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.stacktobasics</groupId>
    <artifactId>spring-boot-nextjs-demo</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>pom</packaging>
    <name>Spring Boot Nextjs Demo</name>
    <modules>
        <module>nextjs-app</module>
        <module>spring-boot-app</module>
    </modules>
    <properties>
        <java.version>17</java.version>
    </properties>

</project>

This quite simply inherits from the Spring Boot parent pom (as is usually the case for Spring Boot applications) and declares two child modules: our Next.js app and our Spring Boot app.

Next.js Maven setup

We'll create a pom.xml file in our nextjs-app folder, which contains the following:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>spring-boot-nextjs-demo</artifactId>
        <groupId>com.stacktobasics</groupId>
        <version>0.0.1-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>nextjs-app</artifactId>

    <build>
        <plugins>
            <plugin>
                <groupId>com.github.eirslett</groupId>
                <artifactId>frontend-maven-plugin</artifactId>
                <version>1.11.3</version>
                <executions>
                    <execution>
                        <id>install node and npm</id>
                        <goals>
                            <goal>install-node-and-npm</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>npm install</id>
                        <goals>
                            <goal>npm</goal>
                        </goals>
                        <phase>generate-resources</phase>
                    </execution>
                    <execution>
                        <id>npm run build</id>
                        <goals>
                            <goal>npm</goal>
                        </goals>
                        <phase>generate-resources</phase>
                        <configuration>
                            <arguments>run build</arguments>
                        </configuration>
                    </execution>
                </executions>
                <configuration>
                    <nodeVersion>v18.14.0</nodeVersion>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

In this maven module we're including the frontend-maven-plugin.

When this module is built this plugin will run the execution steps that we've defined, npm install and npm run build in this case.

If we look in the package.json file generated by Next.js we'll see that npm run build runs next build, which generates all the files required for a production build in the out folder.

Spring Boot Maven setup

Next, let's edit the pom.xml file in the spring-boot-app folder to inherit from our new parent pom, and also to include our Next.js static assets in our Spring Boot application jar. The final pom.xml file will look like this:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <!-- Change parent tag from spring-boot-starter-parent to spring-boot-nextjs-demo -->
    <parent>
        <groupId>com.stacktobasics</groupId>
        <artifactId>spring-boot-nextjs-demo</artifactId>
        <version>0.0.1-SNAPSHOT</version>
    </parent>
    <groupId>com.stacktobasics</groupId>
    <artifactId>spring-boot-app</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>spring-boot-app</name>
    <description>Demo project for Spring Boot that serves Next.js assets</description>
    <properties>
        <java.version>17</java.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <artifactId>maven-resources-plugin</artifactId>
                <version>3.2.0</version>
                <executions>
                    <execution>
                        <id>copy-resources</id>
                        <phase>generate-resources</phase>
                        <goals>
                            <goal>copy-resources</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${basedir}/target/classes/static</outputDirectory>
                            <resources>
                                <resource>
                                    <directory>${basedir}/../nextjs-app/out</directory>
                                    <filtering>false</filtering>
                                </resource>
                            </resources>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

Then maven-resources-plugin will copy our Next.js static assets created by our Next.js maven module, into the static folder in our jar.

Enable static export

Finally, we need to enable the exporting of our Next.js application to HTML. To do this, edit the build script in package.json to include next export:

"scripts": {
    "dev": "next dev",
    "build": "next build && next export",
    "start": "next start",
    "lint": "next lint"
},

Now, let's build the application!

Note : If you don't have Maven installed then copy the .mvn folder and the mvnw file (or mvnw.cmd file if you're on windows) to the root directory, spring-boot-nextjs-demo.

From the root directory, run mvnw clean install.

Oops! The build failed!

You will see something like:

[INFO] info - using build directory: D:\Documents\Development\spring-boot-nextjs-demo\nextjs-app\.next
[INFO] info - Copying "static build" directory
[INFO] info - No "exportPathMap" found in "D:\Documents\Development\spring-boot-nextjs-demo\nextjs-app\next.con
fig.js". Generating map from "./pages"
[INFO] Error: Image Optimization using Next.js' default loader is not compatible with `next export`.
[INFO] Possible solutions:
[INFO] - Use `next start` to run a server, which includes the Image Optimization API.
[INFO] - Configure `images.unoptimized = true` in `next.config.js` to disable the Image Optimization API.   
[INFO] Read more: https://nextjs.org/docs/messages/export-image-api
[INFO] at D:\Documents\Development\spring-boot-nextjs-demo\nextjs-app\node_modules\next\dist\export\index.js
:153:23
[INFO] at async Span.traceAsyncFn (D:\Documents\Development\spring-boot-nextjs-demo\nextjs-app\node_modules\
next\dist\trace\trace.js:79:20)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spring Boot Nextjs Demo 0.0.1-SNAPSHOT:
[INFO]
[INFO] Spring Boot Nextjs Demo ............................ SUCCESS [0.406 s]
[INFO] nextjs-app ......................................... FAILURE [17.515 s]
[INFO] spring-boot-app .................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18.369 s
[INFO] Finished at: 2023-02-18T18:08:02Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.11.3:npm (npm run build) on project n
extjs-app: Failed to run task: 'npm run build' failed. org.apache.commons.exec.ExecuteException: Process exited 
with an error: 1 (Exit value: 1) -> [Help 1]

Disabling Next.js' server-side features

The reason we see this error is that there are limitations when using next export. We'll cover this in more detail in the next section, but for now, change your next.config.js file in nextjs-app to the following:

/** @type {import('next').NextConfig} */
const nextConfig = {
  reactStrictMode: true,
  images: {
    unoptimized: true
  },
}

module.exports = nextConfig

Now let's run our build again using mvnw clean install

Hopefully, by the end of the build, you should see:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spring Boot Nextjs Demo 0.0.1-SNAPSHOT:
[INFO]
[INFO] Spring Boot Nextjs Demo ............................ SUCCESS [0.355 s]
[INFO] nextjs-app ......................................... SUCCESS [01:39 min]
[INFO] spring-boot-app .................................... SUCCESS [8.813 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:49 min
[INFO] Finished at: 2023-02-18T18:02:11Z
[INFO] ------------------------------------------------------------------------

Let's test our spring boot app to see if it's rendering our Next.js static assets.

Run java -jar spring-boot-app/target/spring-boot-app-0.0.1-SNAPSHOT.jar.

Once you see c.s.s.SpringBootAppApplication : Started SpringBootAppApplication in 2.334 seconds (process running for 2.847) go to http://localhost:8080/.

You should see the generated HTML from the index.tsx page:

Adding an Endpoint to our API

To make sure that we can reach both our API and our HTML pages let's add a simple controller.

Under spring-boot-app/src/main/java/com.stacktobasics.springbootapp add the following HelloController.java file:

package com.stacktobasics.springbootapp;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class HelloController {
    @GetMapping("/api/hello-world")
    public String sayHello() {
        return "Hello!";
    }
}

Let's rebuild and re-run our application.

Stop the java -jar process if you haven't already and run mvnw clean install, followed by java -jar spring-boot-app/target/spring-boot-app-0.0.1-SNAPSHOT.jar.

Once our Spring Boot app has started, going to http://localhost:8080/ will again show us the landing page.

Going to http://localhost:8080/api/hello-world will return Hello! as expected, proving that Spring Boot is still routing to our API correctly!

Adding another Next.js page

Lastly, let's check that we're able to navigate between different Next.js pages.

Add a new file called hello.tsx under the nextjs-app\pages folder with the following contents:

export default function Home() {
    return (
        <>
            Hello world!
        </>
    )
}

Next, we'll create a link to this page. Open the index.tsx file and import the next/link package:

import Link from "next/link";

Then, add a link to our new page at the bottom:

...
          <div>
            <Link href={"/hello"}>Go to hello page</Link>
          </div>
        </div>
      </main>
    </>
  )
}

We will now see a link to our Hello page at the bottom of the landing page:

Clicking on this link will bring us to our new page:

However, try refreshing the page. You'll see a 404 error:

Huh? What happened?

Routing to pages without .html

When we clicked the link to go to our hello page from our index page the request to change to a different route was handled from the client side, which worked thanks to Next.js's routing.

However, when we navigate directly to /hello we're going straight to the server. Our server does not have a resource that maps to /hello, so it returns a 404.

We do however have a file called /hello.html. If you navigate to localhost:8080/hello.html you will see our hello page.

This isn't great though, we want users to be able to get to our /hello page directly, without the .html extension!

One easy way to do this is to add a Spring MVC controller (Note the use of @Controller, not @RestController!) to map any request which does not have a file extension to the corresponding HTML page.

Add a class called HtmlController containing the following:

package com.stacktobasics.springbootapp;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;

@Controller
public class HtmlController {

    Logger logger = LoggerFactory.getLogger(HtmlController.class);

    @RequestMapping("/{page:^(?!.*[.].*$).*$}")
    public String requestPage(@PathVariable("page") String page) {
        String htmlPage = "/"+page+".html";
        logger.info("forwarding request to {}", htmlPage);
        return htmlPage;
    }
}

Now going to localhost:8080/hello will correctly render our hello page.

We also see the following log, showing that our HtmlController is working:

[nio-8080-exec-2] c.s.springbootapp.HtmlController : forwarding request to /hello.html

Making local development easier

Whilst what we've done above works well for generating a production build it's quite cumbersome for local development.

Instead, I would recommend using next dev for local development and starting your Spring Boot app in the usual way - such as from an IDE like IntelliJ.

Let's test this out!

Run npm run dev (which runs next dev) from the nextjs-app folder and verify that you can load both localhost:3000 and localhost:3000/hello.

Start up your Spring Boot app from your IDE.

You might have guessed a problem we will face with this approach (hint: it's related to port numbers!).

When we've built our application using maven and running java -jar then both our API and our front-end assets are being served from our Spring Boot application on port 8080.

However, when using next dev the front end will be served from Node.js on port 3000.

This can be problematic when making calls to our back-end service from our front-end application.

Let's show this issue with an example.

Firstly, stop both your Next.js and Spring Boot processes.

Secondly, change your hello.tsx file to make a call to the api/hello-world endpoint, like so:

import { useState, useEffect } from 'react'
export default function Home() {

    const [helloResponse, setHelloResponse] = useState<string|null>(null)

    useEffect(() => {
        fetch('/api/hello-world')
            .then((res) => res.text())
            .then((data) => {
                setHelloResponse(data)
            })
    }, [])

    return (
        <>
            Hello world!
            Response from server call to /api/hello-world: {helloResponse}
        </>
    )
}

Run mvnw clean install, followed by java -jar spring-boot-app/target/spring-boot-app-0.0.1-SNAPSHOT.jar.

Then navigate to localhost:8080/hello. This will show the following:

We can see the response is returned from our /api/hello-world call, as expected.

This time, stop your java -jar process, and run your Next.js app using npm run dev and your Spring Boot app from your IDE.

Navigate again to localhost:8080/hello.

This time you'll see an error:

The reason we see this error is that your request is going to localhost:3000/api/hello-world, rather than localhost:8080/api/hello-world.

We can get around this by using Rewrites. To do this we must change our next.config.js file to the following:

/** @type {import('next').NextConfig} */
const nextConfig = {
    reactStrictMode: true,
    images: {
        unoptimized: true
    },
    async rewrites() {
        return [
            {
                source: '/api/:path*',
                destination: 'http://localhost:8080/api/:path*',
            },
        ]
    }
}

module.exports = nextConfig

Here we've added config to rewrite every request matching /api/** to http://localhost:8080/api/**.

Restart the Node server by re-running npm run dev.

You'll now see that navigating to localhost:3000/hello renders the data from localhost:8080/api/hello-world

Limitations of hosting Next.js static assets from Spring Boot

We touched on the limitations of next export briefly when we had to change our next.config.js file to specify that images should not be optimised.

Next.js has a lot of features - some that work on the server-side and some that work on the client-side. When we run next export we no longer have the Node.js server, and thus we can't take advantage of the server-side features of Next.js that rely on a dedicated Node.js server.

It's worth keeping this in mind as there may be features that you want to use that are server-side features. The following features are not supported:

Image Optimization (default loader)
Internationalized Routing
API Routes
Rewrites
Redirects
Headers
Middleware
Incremental Static Regeneration
fallback: true
getServerSideProps

More on this can be read in the official documentation here.

Conclusion

In this post, we have learned how to generate static assets for Next.js and serve them with a Spring Boot back-end application.

We have learned about some of the common pitfalls of this approach and found ways to get around them.

We have also learned the best way to do local development when using this approach by using Next.js Rewrites.

Finally, we have learned about some of the limitations of using the Export feature in Next.js.

All of the code from this post can be found here.

Till next time!

Adding Correlation IDs to Easily Track Down Errors

Shane Jennings — Wed, 13 Jul 2022 20:20:35 +0000

In this post, we'll look at how we can use Spring Cloud Sleuth to add trace IDs to our application to track down calls and exceptions.

Why Would We Want To Use Correlation IDs?

If you've ever worked on an application that has numerous concurrent calls happening you'll know that it's hard to figure out from the logs which message belongs to which call.

Likewise, if you get an exception in your application and you want to find the root cause it can be hard to know where to look in the logs.

We can use correlation IDs to correlate our log messages and exception stack traces to specific calls in our web application. As well, if we include correlation IDs in our error responses sent back to clients we can use those IDs to trace back to the cause of the error.

One way we can easily add correlation IDs to our application is Spring Cloud Sleuth.

Spring Cloud Sleuth

Spring Cloud Sleuth is part of the wider Spring Cloud library and provides the configuration required for distributed tracing.

Distributed tracing is a great tool for tracing your calls through various microservices, however, the part that we're interested in here is this:

Adds trace and span ids to the Slf4J MDC, so you can extract all the logs from a given trace or span in a log aggregator.

The trace ID is what we will use as our correlation ID.

By default, Spring Cloud Sleuth will add the trace ID as a header when calling other services, which is how the distributed trace is tracked. However, it does not automatically add the trace id to responses from our application. This can easily be added with a filter, which we'll show below.

Setting Up A Spring Boot App with Sleuth

Adding a Simple Controller and Service

Let's start by creating a Spring Boot application with a simple controller and service.

Pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>Blogging-correlation-ids</artifactId>
    <version>1.0-SNAPSHOT</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.7.1</version>
        <relativePath/>
    </parent>

    <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
        <java.version>17</java.version>
        <spring-cloud.version>2021.0.3</spring-cloud.version>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-sleuth</artifactId>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
    </dependencies>

</project>

StackToBasicsApplication

package com.stacktobasics;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class StackToBasicsApplication {
    public static void main(String[] args) {
        SpringApplication.run(StackToBasicsApplication.class, args);
    }
}

HelloWorldController

package com.stacktobasics;

import lombok.extern.slf4j.Slf4j;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping
@Slf4j
public class HelloWorldController {

    private final HelloWorldService helloWorldService;

    public HelloWorldController(HelloWorldService helloWorldService) {
        this.helloWorldService = helloWorldService;
    }

    @GetMapping("/hello")
    public ResponseEntity<String> sayHello() {
        log.info("Someone called the /hello endpoint");
        return ResponseEntity.ok(helloWorldService.sayHello());
    }
}

HelloWorldService

package com.stacktobasics;

import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;

@Component
@Slf4j
public class HelloWorldService {

    public String sayHello() {
        log.info("Returning hello from service");
        return "hello";
    }
}

When we launch this application and call GET /hello we see the following:

2022-07-13 18:30:39.545 INFO [,10f5c8744d01e2d4,10f5c8744d01e2d4] 19460 --- [nio-8080-exec-6] com.stacktobasics.HelloWorldController : Someone called the /hello endpoint
2022-07-13 18:30:39.545 INFO [,10f5c8744d01e2d4,10f5c8744d01e2d4] 19460 --- [nio-8080-exec-6] com.stacktobasics.HelloWorldService : Returning hello from service

Notice the 10f5c8744d01e2d4 ID in our logs. This is repeated twice, once for the span ID and once for the trace ID. In this case they are the same value. We can see that the ID is injected into both our controller and our service logs. This means we can trace the call through whatever classes are used!

If we call GET /hello again we can see we get a different ID:

2022-07-13 18:32:51.910 INFO [,4a6379092ac32a93,4a6379092ac32a93] 19460 --- [nio-8080-exec-7] com.stacktobasics.HelloWorldController : Someone called the /hello endpoint
2022-07-13 18:32:51.911 INFO [,4a6379092ac32a93,4a6379092ac32a93] 19460 --- [nio-8080-exec-7] com.stacktobasics.HelloWorldService : Returning hello from service

Adding the Correlation ID to Exception Responses

It would be useful to be able to return the trace ID when sending back error responses. That way if a user of the application tells us about an error they are experiencing, we can ask them for the response they received and look for the trace ID in our logs.

HttpExceptionHandler

package com.stacktobasics;

import lombok.extern.slf4j.Slf4j;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.ResponseStatus;
import org.springframework.web.bind.annotation.RestControllerAdvice;

@RestControllerAdvice
@Slf4j
public class HttpExceptionHandler {
    private final CorrelationIDHandler correlationIDHandler;

    public HttpExceptionHandler(CorrelationIDHandler correlationIDHandler) {
        this.correlationIDHandler = correlationIDHandler;
    }

    @ExceptionHandler(IllegalArgumentException.class)
    @ResponseStatus(HttpStatus.BAD_REQUEST)
    public ResponseEntity<ExceptionResponse> handleIllegalArgumentException(IllegalArgumentException exception) {
        log.warn(exception.getMessage(), exception);
        var exceptionResponse = new ExceptionResponse(HttpStatus.BAD_REQUEST, exception, correlationIDHandler.getCorrelationId());
        return new ResponseEntity<>(exceptionResponse, exceptionResponse.getStatus());
    }
}

CorrelationIDHandler

package com.stacktobasics;

import org.springframework.cloud.sleuth.Span;
import org.springframework.cloud.sleuth.TraceContext;
import org.springframework.cloud.sleuth.Tracer;
import org.springframework.stereotype.Component;

import java.util.Optional;

@Component
public class CorrelationIDHandler {
    private final Tracer tracer;

    public CorrelationIDHandler(Tracer tracer) {
        this.tracer = tracer;
    }

    public String getCorrelationId() {
        return Optional.of(tracer).map(Tracer::currentSpan).map(Span::context).map(TraceContext::traceId).orElse("");
    }
}

ExceptionResponse

package com.stacktobasics;

import com.fasterxml.jackson.annotation.JsonFormat;
import lombok.Getter;
import lombok.NonNull;
import org.springframework.http.HttpStatus;

import java.time.LocalDateTime;

@Getter
public class ExceptionResponse {
    private final String correlationId;
    private final HttpStatus status;
    @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd HH:mm:ss")
    private final LocalDateTime datetime;
    private final String error;

    ExceptionResponse(@NonNull HttpStatus status, @NonNull Exception ex, String correlationId) {
        datetime = LocalDateTime.now();
        this.status = status;
        this.error = ex.getMessage();
        this.correlationId = correlationId;
    }
}

Let's add a method to our controller and service to trigger an exception.

HelloWorldController

...

@RestController
@RequestMapping
@Slf4j
public class HelloWorldController {

    ...

    @GetMapping("/bad-call")
    public void badCall() {
        log.info("Someone called the /bad-call endpoint");
        helloWorldService.fakeBadCall();
    }
}

**HelloWorldService

...

@Component
@Slf4j
public class HelloWorldService {

    ...

    public void fakeBadCall() {
        log.info("About to throw IllegalArgumentException...");
        throw new IllegalArgumentException("Exception from Hello World Service");
    }
}

Things to notice are:

CorrelationIDHandler uses the injected Tracer object to get a copy of the trace ID.
ExceptionResponse is the response object we return to users when we encounter an exception. It contains the trace ID as a correlation ID field.
We have an exception handler HttpExceptionHandler which handles uncaught exceptions and provides and appropriate response to the user. Here we are only handling IllegalArgumentException.

When we call GET /bad-call we see the following in our logs:

2022-07-13 18:47:20.751 INFO [,b490ea4b7230260a,b490ea4b7230260a] 14716 --- [nio-8080-exec-5] com.stacktobasics.HelloWorldController : Someone called the /bad-call endpoint
2022-07-13 18:47:20.751 INFO [,b490ea4b7230260a,b490ea4b7230260a] 14716 --- [nio-8080-exec-5] com.stacktobasics.HelloWorldService : About to throw IllegalArgumentException...
2022-07-13 18:47:20.755 WARN [,b490ea4b7230260a,b490ea4b7230260a] 14716 --- [nio-8080-exec-5] com.stacktobasics.HttpExceptionHandler : Exception from Hello World Service

java.lang.IllegalArgumentException: Exception from Hello World Service
    at com.stacktobasics.HelloWorldService.fakeBadCall(HelloWorldService.java:17) ~[classes/:na]
    at com.stacktobasics.HelloWorldController.badCall(HelloWorldController.java:29) ~[classes/:na]
    ...

We also see the following response:

{
"correlationId": "b490ea4b7230260a",
"status": "BAD_REQUEST",
"datetime": "2022-07-13 18:47:20",
"error": "Exception from Hello World Service"
}

We can see that the correlationID in the response matches the trace ID in the logs.

Adding the Trace ID to the Header

Having the trace ID is great for tracing calls through our logs. However, if we want to track the trace ID in another application that doesn't use Spring Sleuth (such as a Node.JS application) then we need to find a way of exposing the trace ID to callers.

This can easily be done by adding a filter to our application which adds the trace ID as a header.

TraceFilter

@Component
public class TraceFilter implements Filter {

    private static final String TRACE_ID_HEADER_NAME = "X-B3-TraceId";
    private final CorrelationIDHandler correlationIDHandler;

    public TraceFilter(CorrelationIDHandler correlationIDHandler) {
        this.correlationIDHandler = correlationIDHandler;
    }

    @Override
    public void doFilter(ServletRequest req, ServletResponse res,
                         FilterChain chain) throws IOException, ServletException {

        HttpServletResponse response = (HttpServletResponse) res;
        if (!response.getHeaderNames().contains(TRACE_ID_HEADER_NAME)) {
            var id = correlationIDHandler.getCorrelationId();
            if (!id.isEmpty()) response.setHeader(TRACE_ID_HEADER_NAME, id);
        }
        chain.doFilter(req, res);
    }
}

This filter adds the X-B3-TraceId header to any response that does not already contain a trace ID. The X-B3-TraceId header is what Sleuth adds when sending requests to other services to keep track of the trace through microservices, so we've used the same name to be consistent.

If we run our application again and call GET /hello we will find the trace ID has been added to the response headers:

HTTP/1.1 200
X-B3-TraceId: c9ac548611166be7

Conclusion

In this post, we've learned how to use Spring Cloud Sleuth to trace calls through our application. We've also learned how to include those trace IDs in error responses sent back to the client, as well as including the trace ID as an HTTP header to propagate it through other clients.

Till next time!

Using Jackson Subtypes to Write Better Code

Shane Jennings — Fri, 29 Apr 2022 15:54:36 +0000

In this post, we'll look at how Jackson Subtypes can automatically map JSON to the right subclass, without having to write your own converter.

First, we'll give some background into why this is useful, and then we'll go through some practical examples for two different use cases.

Why are Jackson Subtypes Useful?

You'll sometimes find that you receive JSON HTTP responses that look similar but have a few fields which are different. If you have to map these to Java classes then it can be annoying to have to replicate all of the common fields across each of your DTO classes.

Let's look at an example of the problem.

Let's say you get a response from a service that looks like this:

[
  {
    "id": "1",
    "type": "onlineOrder",
    "quantity": 1,
    "order": {
      "name": "Playstation 5",
      "onlineCode": "123456",
      "referralId": "6789",
      "customerId": "5"
    }
  },
  {
    "id": "2",
    "type": "shopOrder",
    "quantity": 3,
    "order": {
      "name": "Xbox One",
      "shopId": "2",
      "paymentType": "card"
    }
  }
]

We can see that the two objects in the list look similar, but both have a different order object format.

We could map these to DTO objects using the following class:

public class GenericOrderDTO {
    private String id;
    private String type;
    private int quantity;
    private Object order;

    // getters and setters omitted
}

This will work, but it won't give us any indication about what is inside the order object, plus we'll have to do some casting.

We could also use Map<String, String> instead to represent the order object, which will give us a map of field name to value.

In either case, we would need to write our own mapper to figure out if the order is a shop type or an online type and map the fields accordingly.

A better way to do this is to use Jackson's @JsonTypeInfo and @JsonSubTypes annotations with inheritance.

How Jackson Subtypes Work

Jackson has a concept of subtypes, which is similar to a subclass.

With inheritance you have your parent class which contains common fields across each of your subclasses, and your subclasses which contain fields unique to them. Subtypes work in a similar way, with the help of some annotations.

In order to use subtypes we need to have some property in the JSON that tells us which subtype to use. In our example above we have the type field, which states whether the order object is an online order or a shop order. Another way is when the type field is inside the subtype itself.

Let's look at both.

Type Field is on the Root Object

Using our example above, we can map the JSON to the following DTOs:

import com.fasterxml.jackson.annotation.JsonTypeInfo;

public class OrderDTO {
    private String id;
    private int quantity;
    private String type;
    @JsonTypeInfo(
            use = JsonTypeInfo.Id.NAME,
            include = JsonTypeInfo.As.EXTERNAL_PROPERTY,
            property = "type",
            defaultImpl = UnknownTypeOrderDTO.class
    )
    private OrderTypeDTO orderTypeDTO;

    // getters and setters omitted
}


import com.fasterxml.jackson.annotation.JsonSubTypes;

@JsonSubTypes({
        @JsonSubTypes.Type(value = OnlineTypeOrderDTO.class, name = "onlineOrder"),
        @JsonSubTypes.Type(value = ShopTypeOrderDTO.class, name = "shopOrder")
})
public class OrderTypeDTO {
}


public class OnlineTypeOrderDTO extends OrderTypeDTO {
    private String name;
    private String referralId;
    private String customerId;

    // getters and setters omitted
}


public class ShopTypeOrderDTO extends OrderTypeDTO {
    private String name;
    private String shopId;
    private String paymentType;

    // getters and setters omitted
}


import com.fasterxml.jackson.annotation.JsonIgnoreProperties;

@JsonIgnoreProperties(ignoreUnknown = true)
public class UnknownTypeOrderDTO extends OrderTypeDTO {

}

Looking at OrderTypeDTO first, we have added an annotation that lists what class should map to what name.

In the OrderDTO class, we include an annotation that tells Jackson to map the orderTypeDTO to the class where the value of the type field matches the name specified in the @JsonSubTypes annotation.

We also state that the type field is an external property. That is, it's not a field on OrderTypeDTO, but rather on OrderDTO.

Finally, we state that if Jackson can't find a matching class based on the value of type, then use UnknownTypeOrderDTO instead. This is protect us from exceptions where the order endpoint might add a new type before we change our code. You can then have some code that appropriately deals with any orders that map to UnknownTypeOrderDTO.

Type field is on the Subtype Object

Another way is when the type field is on the subtype object itself.

Let's change our example slightly such that the JSON we receive looks like this:

[
  {
    "id": "1",
    "quantity": 1,
    "order": {
      "name": "Playstation 5",
      "type": "onlineOrder",
      "onlineCode": "123456",
      "referralId": "6789",
      "customerId": "5"
    }
  },
  {
    "id": "2",
    "quantity": 3,
    "order": {
      "name": "Xbox One",
      "type": "shopOrder",
      "shopId": "2",
      "paymentType": "card"
    }
  }
]

In this case, our DTOs would look like this:

import com.fasterxml.jackson.annotation.JsonTypeInfo;

public class OrderDTO {
    private String id;
    private int quantity;
    @JsonTypeInfo(
            use = JsonTypeInfo.Id.NAME,
            property = "type",
            defaultImpl = UnknownTypeOrderDTO.class
    )
    private OrderTypeDTO orderTypeDTO;

    // getters and setters omitted
}


import com.fasterxml.jackson.annotation.JsonSubTypes;

@JsonSubTypes({
        @JsonSubTypes.Type(value = OnlineTypeOrderDTO.class, name = "onlineOrder"),
        @JsonSubTypes.Type(value = ShopTypeOrderDTO.class, name = "shopOrder")
})
public class OrderTypeDTO {
    private String type;
}

The ShopTypeOrderDTO, OnlineTypeOrderDTO, and UnknownTypeOrderDTO classes are exactly the same as before.

The difference here is that the type field has moved to the OrderTypeDTO class, and include = JsonTypeInfo.As.EXTERNAL_PROPERTY was removed from the @JsonTypeInfo annotation.

Conclusion

In this post, we've expressed when and why Subtypes are useful, as well as learned how to use them when you have the type inside or outside of the subclass through practical examples.

Till next time!

13 Reasons To Move From Java 8 To Java 11

Shane Jennings — Sat, 16 May 2020 15:40:35 +0000

Despite Java 11 being released in September 2018, 64% of developers report that Java 8 remains the most often used release.

A further 27% of people said that there are "no features you need in later versions".

In this blog post, I will go over why I think this is wrong by showing some of the new exciting features post Java 8!

Changes to the Release Model

Before we get started, let's discuss briefly the changes to Java's release model.

In the past, Java hasn't had particularly fast release times. Java 6 came out in December 2006, Java 7 wasn't released until July 2011, and Java 8 wasn't released until March 2014! These were big bang changes and followed a waterfall pattern. If a feature didn't make it into the release you could be waiting quite a few years before you'd see it.

This is something Oracle has changed with Java 9 going forward. Releases now have a 6-month cadence, with a Long Term Support (LTS) release happening every 3 years. This is why you hear people talk about Java 11 rather than 9, 10, or 12. Java 11 was the first release with LTS since Java 8. The next release with LTS will be Java 17.

Whilst you probably won't migrate to the versions in between Java 11 and Java 17 (at least not on enterprise applications), you can definitely check out the JDKs in between and play around with the features!

Java 9

No More Java EE

JavaEE is no longer part of the core JDK. JavaEE included features such as:

XML Bindings
JPA
Dependency Injection

This means that the JDK is smaller, and frees up Oracle to focus on features that improve the language, rather than the surrounding tools.

When you migrate from Java 8 to Java 11, most likely this will be your biggest pain point. You'll see errors such as ClassNotDef or MethodNotFound. There are ways to get around this without making many changes, but it's still a pain.

So what has happened to all of the Java EE features?

Eclipse is now handling them! JavaEE has been rebranded as Jakarta EE, and can be read about here. This is a good thing, as Eclipse is focusing on making Java a Cloud Native platform, something that Oracle couldn't keep up with.

A Better HTTP Client

Not many people use the default HttpURLConnection API in Java. Most people will use Apache HttpClient, Jetty, or RestTemplate. This is because HttpUrlConnection isn't very user-friendly.

However, in Java 9, a new HTTP client was included which is much easier to use.

The new API has 3 core classes:

HttpRequest
HttpClient
HttpResponse

The new HTTP client supports:

both synchronous and asynchronous requests
HTTP/2 and HTTP/1.1 compatibility

HTTP/2 is a big step up from HTTP/1.1, which was last released in 1997. HTTP/3 is getting more support now as well, so hopefully we should see support for this in Java in the future too.

Here's an example of setting up an HTTP client using HTTP2:

HttpClient client = HttpClient.newBuilder()
      .version(Version.HTTP_2)
      .build();

and to send a request asynchronously:

HttpRequest request = HttpRequest.newBuilder()
      .uri(URI.create("http://openjdk.java.net/"))
      .build();
client.sendAsync(request, BodyHandlers.ofString())
      .thenApply(HttpResponse::body)
      .thenAccept(System.out::println)
      .join();

Simple, right?

Finally a REPL

REPL = R ead E val P rint L oop

A lot of other languages have REPLs and now finally Java does too, with its REPL called jShell. If you don't know what a REPL is, Python is a good example. You can type python into your command prompt to start the language shell and execute each line you write.

The Java REPL is most useful for seeing the output of small amounts of code, or playing with a new feature.

Why is jShell easier than creating your own class and experimenting with an IDE?

you don't have to write complete Java code
You don't need to declare classes if you don't want to
you don't need to include sem-colons
You can define variables, methods, classes, imports, and expressions, and you can also completely overwrite any of these.

It's also really easy to use:

> jshell
| Welcome to JShell -- Version 11
| For an introduction type: /help intro
jshell> String myStr = "hello stack to basics!"
myStr ==> "hello stack to basics!"

jshell> myStr
myStr ==> "hello stack to basics!"

jshell> myStr + " How are you?"
$3 ==> "hello stack to basics! How are you?"

jshell> $3
$3 ==> "hello stack to basics! How are you?"

This article here shows some of the things you can do with jShell.

Emojis4J

Unicode 7 is here in Java, which means you can finally use emojis! (This feature is not really called Emojis4J, but I wish it was).

We have to be careful here though. Lets look at an a exmaple:

public class Emojis4J {
    public static void main(String[] args) {
        String bear = "🐻";
        char bearChar = bear.charAt(0);

        String aString = "a";
        char aChar = aString.charAt(0);

        System.out.println("bear: " + bear)
        System.out.println("bearChar: " + aChar);
        System.out.println("a: " + bearChar);
    }
}

This prints out:

bear: 🐻
bearChar: ?
a: a

The bear char didn't print properly. Why not?

This is because emojis are 5 digit hex numbers. Each hex number needs 4 bits, therefore a 5 digit hex number needs 5x4=20 bits, with the bear emoji represented by \u1f43b. A java character is 2 bytes=2x8 bits = 16 bits. Not big enough to hold an emoji!

To get around this you can represent emojis with two 16 bit hexadecimal values. The bear is represented with the string \ud83d\udc3b.

These are called surrogate pairs, and are used to solve the problem in languages where chars are not big enough to hold some unicode characters.

We can convert the bear to 2 chars properly like so:

public class Emojis4J {
    public static void main(String[] args) {
        String bear = "🐻";
        int bearCodepoint = bear.codePointAt(bear.offsetByCodePoints(0, 0));
        char[] bearSurrogates = {Character.highSurrogate(bearCodepoint),
                Character.lowSurrogate(bearCodepoint)};
        System.out.println("Value of bearSurrogates: " +
                String.valueOf(bearSurrogates));
    }
}

This prints:

Value of bearSurrogates: 🐻

That's better.

Smarter Strings

In Java strings are stored as an array of chars, with each char taking 2 bytes. However, research shows two things:

strings are a major component of heap usage
most String objects contain only Latin-1 characters.

Latin-1 characters only require 1 byte.

This feature adds an encoding flag to Strings which, based on the content of the string, will use either two bytes per character or one byte.

This can save you a lot of space!

Factory methods for collections

Finally, java offers a standard way to instantiate collections.

We also now have immutable collections which implement the Collection interface but don't implement any mutable methods.

public class JavaCollections {
    public static void main(String[] args) {
        //Immutable Collections:
        // Empty List
        List.of();

        // Easy initialisation of list, array-based List 
        // implementation (if more than 2 elements, check out the docs )
        final List<String> immutableList = List.of("String1", "String2");
        // The line below would throw the exception: Exception in thread 
        // "main" java.lang.UnsupportedOperationException:
        // immutableList.add("String3");

        // Empty map
        Map.of();

        // Easy initialisation of map, array-based Map implementation
        Map.of("My key1", 1, "My Key2", 2);

        // Mutable Collections
        // Not great, this way might end up creating up to 3 arrays
        List<String> mutableList = new ArrayList<>(List.of("String1"));

        // This will work
        mutableList.add("String4");
    }
}

Experimenting with Ahead-of-Time compilation

Another new feature of Java is Ahead-of-Time compilation.

To understand why this is an exciting addition, let's look at the way most JVMs roughly work:

byte code is executed by the JVM
The JVM will either:

This is great, but it can mean that for large programs the JIT can take a long time to "warm up", and code may never be compiled at all (so it will execute slower).

Ahead-of-Time compilation allows us to compile Java classes to native code prior to launching the JVM. This would mean that our program should run a lot faster, as there is no interpretation or compilation happening whilst the program is running.

As an example, let's try and compile the following class:

public class MyClassForAOT {
    public static void main(String[] args) {
        System.out.println("Hello from StackToBasics");
    }
}

We can compile this using AOT like so:

>javac MyClassForAOT.java
>jaotc --output MyClassForAOT.so MyClassForAOT.class
>java -XX:AOTLibrary=./MyClassForAOT.so MyClassForAOT
Hello from StackToBasics

AOT compilation could be particularly useful for cloud service, especially when using AWS Lambdas or Google's Cloud Functions, where faster start-up time and having a smaller code footprint is important.

A lot more can be said on AOT compilation, but that shall be saved for a separate post 😃

Java 10

Finally, the var keyword

Type inference is now extended to the declaration of local variables with initialisers.

public class TypeInference {

    public static void main(String[] args) {
        var hello = "Hello Stack to Basics!";
        // hello = 5; This will not work, hello's type has been inferred as string, it cannot be changed to int
        System.out.println(hello);
    }
}

A misconception is that type inference == dynamically typed. This is not true. Each variable still requires a type, it is just that the type can be inferred rather than explicitly stated. Java is still a statically-typed language.

Remember that type inference already existed in Java, when using lambdas. We can see below that we did not need to declare the type for b, rather it was inferred.

int maxWeight = blocks.steam()
    .filter(b -> b.getColor() == BLUE)
    .mapToInt(Block::getWeight)
    .max();

Dogfooding Their Own Language

Before Java 9, the JIT compiler was written in C++. Not just C++, but a specific dialect of C++, which made it difficult for developers to make improvements.

Now they've finally switched to a different language and they're using... you guessed it, Java.

This is still in the early experimental stages, but wouldn't it be great to have a JIT compiler that developers can improve?

Var for Lambdas

Following on from local variable type inference, we can now use var in lambdas:

(var s1, var s2) -> s1 + s2

Which is equivalent to the following:

(String s1, String s2) -> s1 + s2 //or
(s1, s2) -> s1 + s2

Why bother with var for lambdas? Well, we can now do things like this which we couldn't before (without explicitly declaring s1 and s2 as strings):

(@Nonnull var s1, @Nullable var s2) -> s1 + s2

Emojis4J2

There are 56 more emojis!

You can now display the bitcoin symbol. How exciting.

Launching Single-File Programs

You can now run single-file java programs without compiling them, with any arguments placed after the file name being treated as program arguments. As example, lets say we have the following class:

public class Factorial {

    public static void main(String[] args) {
        if(args.length != 1) throw new RuntimeException("Please provide 1 number");

        int number = Integer.parseInt(args[0]);
        System.out.println("Answer: " + calculateFactorial(number));
    }

    public static int calculateFactorial(int number) {
        if(number < 0) throw new RuntimeException("number must be positive");

        if(number == 0) return 1;
        if(number == 1) return 1;
        if(number == 2) return 2;
        else return calculateFactorial(number-1) * number;
    }
}

Normally we would have to compile this to byte code and then run it, like so:

>javac Factorial.java
>java Factorial 5
Answer: 120

But now, we can skip out the first step:

>java Factorial.java 5
Answer: 120

Epsilon, The Garbage Collector for AWS Lambda and Google Cloud Functions?

A new gabage collector has been added! ... Which doesn't collect any garbage.

It handles memory allocation but does not implement any actual memory reclamation mechanism. Once the heap is exhausted, the JVM will shut down.

The main usages of this will most likely be for:

testing how your program uses memory
(Extremely) short-lived jobs, such as lambdas.

Conclusion

This wraps up some of my favourite features of Java 9 to 11. I'm excited for the future of Java, and it's great to see some well-requested features finally making it into the JDK. Hopefully, this blog convinces you to explore some of these new features and take the plunge to upgrade to Java 11 if you haven't already.

Happy Coding!

Mockito Cheatsheet

Shane Jennings — Mon, 04 May 2020 15:53:25 +0000

Mockito is a great framework for testing in java. I use it all the time and have done for many years now. It works hand in hand with dependnecy injection, a topic I covered in my last blog "Spring for Humans". However I sometimes find it's a victim of it's own success - there's a lot you can do with it so it's easy to forget how to do things!

So here's a cheat sheet which covers most of the features I use in Mockito.

Setting Up Mockito

Using JUnit <5 Annotations

import org.mockito.junit.MockitoJUnitRunner;
import org.junit.runner.RunWith;

@RunWith(MockJUnitRunner.class)
public class MyClassTest {
    ...
}

Using JUnit 5 Annotations

import org.mockito.junit.jupiter.MockitoExtension;
import org.junit.jupiter.api.extension.ExtendWith;

@ExtendWith(MockitoExtension.class)
public class MyClassTest {
    ...
}

Using initMocks

Useful if you need more than one RunWith/Extend (e.g. SpringJunit4ClassRunner)

import org.mockito.MockitoAnnotations;
import org.junit.Before;

public class MyClassTest {
    @Before
    public void before() throw Exception {
        Mockito.initMocks(this);
    }
    ...
}

Creating Mocks and Spies

An easy way I use to remember the difference between mocks and spies is:

Mock: By default, all methods are stubbed unless you say otherwise.
Spy: By default, all methods use real implementation unless you say otherwise.

Creating a Mock

Use Annotations on Fields

There seems to be a consensus that this is the cleanest way to mock an object (which goes against my brain's aversion for field injection, but that's a story for another day).

import org.mockito.MockitoAnnotations;
import org.junit.Before;
import org.mockito.Mock;

public class MyClassTest {

    @Mock
    private MyClass myObject;

    @Before
    public void before() throw Exception {
        Mockito.initMocks(this);
        // myObject is now a mock
    }
    ...
}

Create from Method

import org.mockito.MockitoAnnotations;
import org.junit.Before;
import org.mockito.Mock;

public class MyClassTest {

    private MyClass myObject;

    @Before
    public void before() throw Exception {
        Mockito.initMocks(this);
        myObject = Mockito.mock(MyClass.class);
    }
    ...
}

Create from Method using Initialised Object

Useful if you want a mock to be created from an object which has been constructed using a constructor.

import org.mockito.MockitoAnnotations;
import org.junit.Before;
import org.mockito.Mock;

public class MyClassTest {

    private Person person = Person("name", 18);

    @Before
    public void before() throw Exception {
        Mockito.initMocks(this);
        person = Mockito.mock(person);
    }
    ...
}

Creating a Spy

This is very similar to mocks.

Use Annotations on Fields

import org.mockito.MockitoAnnotations;
import org.junit.Before;
import org.mockito.Spy;

public class MyClassTest {

    @Spy
    private MyClass myObject;

    @Before
    public void before() throw Exception {
        Mockito.initMocks(this);
        // myObject is now a spy
    }
    ...
}

Create from Method

import org.mockito.MockitoAnnotations;
import org.junit.Before;
import org.mockito.Spy;

public class MyClassTest {

    private MyClass myObject;

    @Before
    public void before() throw Exception {
        Mockito.initMocks(this);
        myObject = Mockito.spy(MyClass.class);
    }
    ...
}

Create from method using Initialised Object

Useful if you want a spy to be created from an object which has been constructed using a constructor.

import org.mockito.MockitoAnnotations;
import org.junit.Before;
import org.mockito.Spy;

public class MyClassTest {

    private Person person = Person("name", 18);

    @Before
    public void before() throw Exception {
        Mockito.initMocks(this);
        person = Mockito.spy(person);
    }
    ...
}

Injecting Mocks

Using @InjectMocks Annotation

public class InjectTest {

    @Mock
    private BlogRepository blogRepository;
    @InjectMocks
    private BlogPostService blogPostService;

    @Before
    public void setup() {
        MockitoAnnotations.initMocks(this);
        System.out.println(blogPostService);
    }
}

Using Constructor

public class InjectTest {

    @Mock
    private BlogRepository blogRepository;
    @InjectMocks
    private BlogPostService blogPostService;

    @Before
    public void setup() {
        MockitoAnnotations.initMocks(this);
        blogPostService = new BlogPostService(blogRepository);
    }
}

Stubbing a method

Returning from a stubbed Method

@Test
public void getAllBlogPosts_withBlogPostsInDb_returnsBlogPosts() {
    // arrange
    List<BlogPost> expected = Arrays.asList(new BlogPost("Spring for humans"), 
            new BlogPost("Mockito Cheatsheet"));
    BlogRepository blogRepository = Mockito.mock(BlogRepository.class);
    Mockito.when(blogRepository.getAllBlogPosts()).thenReturn(expected);
    BlogPostService service = new BlogPostService(blogRepository);

    // act
    List<BlogPost> actual = service.getAllBlogPosts();

    // assert
    assertEquals(expected, actual);
}

Providing an Alternative Implementation for a Method

@Test
public void getBlogPostById_withExistingBlogPostInDb_returnsBlogPost() {
    // arrange
    BlogPost expected = new BlogPost("Spring for Humans");
    BlogRepository blogRepository = Mockito.mock(BlogRepository.class);
    Mockito.when(blogRepository.getBlogPostById(Mockito.anyInt())).thenAnswer(invocationOnMock -> {
        int id = invocationOnMock.getArgument(0);
        if(id == 1) return Optional.of(expected);
        else return Optional.empty();
    });
    BlogPostService service = new BlogPostService(blogRepository);

    // act
    BlogPost actual = service.getBlogPostById(1).get();

    // assert
    assertEquals(expected, actual);
}

Throwing an Exception from a Method

@Test(expected = IllegalArgumentException.class)
public void getBlogPostById_withMissingBlogPost_throwsException() {
    // arrange
    BlogPost expected = new BlogPost("Spring for Humans");
    BlogRepository blogRepository = Mockito.mock(BlogRepository.class);
    Mockito.when(blogRepository.getBlogPostById(1)).thenThrow(new IllegalArgumentException());
    BlogPostService service = new BlogPostService(blogRepository);

    // act
    BlogPost actual = service.getBlogPostById(1).get();

    // assert
    // Test will pass if exception of type IllegalArgumentException is thrown
}

Calling the Real Implementation

@Test
public void getBlogPostById_withMissingBlogPost_returnsEmpty() {
    // arrange
    BlogRepository blogRepository = Mockito.mock(BlogRepository.class);
    Mockito.when(blogRepository.getBlogPostById(1)).thenCallRealMethod();
    BlogPostService service = new BlogPostService(blogRepository);

    // act
    Optional<BlogPost> actual = service.getBlogPostById(1);

    // assert
    assertFalse(actual.isPresent());
}

Using Deep Stubs

Deep stubs are used for methods which are chained, but you don't care about the intermediate values. These should be used sparingly, as discussed in the Mockito documentation.

// Without deep stubs
@Test
public void checkIfDriverIsPostgres_withPostgresDriver_returnsTrue() {
    // arrange
    Configuration configMock = Mockito.mock(Configuration.class);
    RepositoryConfiguration repositoryConfigMock = Mockito.mock(RepositoryConfiguration.class);
    Mockito.when(configMock.getRepositoryConfiguration()).thenReturn(repositoryConfigMock);
    DatabaseConfiguration databaseConfigMock = Mockito.mock(DatabaseConfiguration.class);
    Mockito.when(repositoryConfigMock.getDatabaseConfiguration()).thenReturn(databaseConfigMock);
    DatabaseConfiguration.DriverConfiguration driverConfigMock = Mockito.mock(DatabaseConfiguration.DriverConfiguration.class);
    Mockito.when(databaseConfigMock.getDriverConfiguration()).thenReturn(driverConfigMock);
    Mockito.when(driverConfigMock.getDriverType()).thenReturn("postgres");

    BlogRepository blogRepository = new BlogRepository(configMock);

    // act
    boolean actual = blogRepository.checkIfDriverIsPostgres();

    // assert
    Assert.assertTrue(actual);
}

// With deep stubs
@Test
public void usingDeepStubs_checkIfDriverIsPostgres_withPostgresDriver_returnsTrue() {
    // arrange
    Configuration configMock = Mockito.mock(Configuration.class, Mockito.RETURNS_DEEP_STUBS);
    Mockito.when(configMock.getRepositoryConfiguration()
            .getDatabaseConfiguration().getDriverConfiguration().getDriverType()).thenReturn("postgres");

    BlogRepository blogRepository = new BlogRepository(configMock);

    // act
    boolean actual = blogRepository.checkIfDriverIsPostgres();

    // assert
    Assert.assertTrue(actual);
}

Stubbing a Void Method

Providing an Alternative Implementation for a Method

@Test
public void getAllBlogPosts_withBlogPostsInDb_callsVerifyConnectionToDatabaseIsAlive() {
    // arrange
    Configuration configMock = Mockito.mock(Configuration.class);
    BlogRepository blogRepository = new BlogRepository(configMock);

    AtomicBoolean verifyMethodCalled = new AtomicBoolean(false);
    Mockito.doAnswer(invocationOnMock -> {
        // We can do whatever we want here, and it will be executed when
        // verifyConnectionToDatabaseIsAlive
        // If the method had any arguments, we can capture them here
        verifyMethodCalled.set(true);
        return null;
    }).when(configMock).verifyConnectionToDatabaseIsAlive();

    // act
    blogRepository.getAllBlogPosts();

    // assert
    assertTrue(verifyMethodCalled.get());
}

Throwing an Exception from a Method

@Test(expected = DatabaseDownException.class)
public void getAllBlogPosts_withConnectionToDbDown_throwsException() {
    // arrange
    Configuration configMock = Mockito.mock(Configuration.class);
    BlogRepository blogRepository = new BlogRepository(configMock);

    AtomicBoolean verifyMethodCalled = new AtomicBoolean(false);
    Mockito.doThrow(new DatabaseDownException()).when(configMock).verifyConnectionToDatabaseIsAlive();

    // act
    blogRepository.getAllBlogPosts();

    // assert
    // Test will pass if exception of type DatabaseDownException is thrown
}

Calling the Real Implementation

@Test
public void getAllBlogPosts_withBlogPostsInDb_returnsBlogPosts() {
    // arrange
    Configuration configMock = Mockito.mock(Configuration.class);
    BlogRepository blogRepository = new BlogRepository(configMock);

    AtomicBoolean verifyMethodCalled = new AtomicBoolean(false);
    Mockito.doCallRealMethod().when(configMock).verifyConnectionToDatabaseIsAlive();

    // act
    blogRepository.getAllBlogPosts();

    // assert
    // Test will pass if exception of type DatabaseDownException is thrown
}

Matchers

Using a Real Parameter

@Test
public void getBlogPostByAuthorAndAfterDate_withExistingBlogPostsInDb_returnsBlogPosts() {
    // arrange
    List<BlogPost> expected = Collections.singletonList(new BlogPost("Spring for Humans"));
    BlogRepository blogRepository = Mockito.mock(BlogRepository.class);
    Date date = new Date();
    String author = "Shane";
    Mockito.when(blogRepository.getBlogPostsByAuthorAndAfterDate(author, date)).thenReturn(expected);
    BlogPostService service = new BlogPostService(blogRepository);

    // act
    List<BlogPost> actual = service.getBlogPostsByAuthorAndAfterDate(author, date);

    // assert
    assertEquals(expected, actual);
}

Using any()

@Test
public void getBlogPostByAuthorAndAfterDate_withoutMatchingBlogPostsInDb_returnsEmptyList() {
    // arrange
    BlogRepository blogRepository = Mockito.mock(BlogRepository.class);
    Mockito.when(blogRepository.getBlogPostsByAuthorAndAfterDate(Mockito.any(), Mockito.any())).thenReturn(Collections.emptyList());
    BlogPostService service = new BlogPostService(blogRepository);

    // act
    List<BlogPost> actual = service.getBlogPostsByAuthorAndAfterDate("any author", new Date());

    // assert
    assertTrue(actual.isEmpty());
}

Using anyClass()

@Test
public void getBlogPostByAuthorAndAfterDate_withMatchingAuthorAndDate_returnsPostsSortedByDate() {
    // arrange
    BlogPost first = new BlogPost("Spring for Humans", "Shane", new Date(5));
    BlogPost second = new BlogPost("Mockito Cheatsheet", "Shane", new Date(6));
    BlogPost third = new BlogPost("?", "Shane", new Date(7));
    BlogRepository blogRepository = Mockito.mock(BlogRepository.class);
    // Good for overloaded methods, you can specify type of params so the call isn't ambiguous.
    Mockito.when(blogRepository.getBlogPostsByAuthorAndAfterDate(Mockito.any(String.class), Mockito.any(Date.class)))
            .thenReturn(Arrays.asList(second, third, first));
    BlogPostService service = new BlogPostService(blogRepository);

    List<BlogPost> expected = Arrays.asList(first, second, third);

    // act
    List<BlogPost> actual = service.getBlogPostsByAuthorAndAfterDate("any author", new Date());

    // assert
    assertEquals(expected.get(0), actual.get(0));
    assertEquals(expected.get(1), actual.get(1));
    assertEquals(expected.get(2), actual.get(2));
}

Using eq()

@Test
public void getBlogPostByAuthorAndAfterDate_withMatchingAuthorButFutureDate_returnsEmptyList() {
    // arrange
    List<BlogPost> expected = Collections.singletonList(new BlogPost("Spring for Humans"));
    BlogRepository blogRepository = Mockito.mock(BlogRepository.class);
    Date date = new Date();
    Mockito.when(blogRepository.getBlogPostsByAuthorAndAfterDate(Mockito.any(), Mockito.eq(date))).thenReturn(expected);
    BlogPostService service = new BlogPostService(blogRepository);

    // act
    List<BlogPost> actual = service.getBlogPostsByAuthorAndAfterDate("any author", date);

    // assert
    assertTrue(actual.isEmpty());
}

Using Custom Matcher

@Test
public void checkIfBlogPostHasBeenSaved_withBlogPost_returnsTrue() {
    // arrange
    BlogPost post = new BlogPost("Spring for Humans", "Shane", new Date(5));
    BlogRepository blogRepository = Mockito.mock(BlogRepository.class);

    ArgumentMatcher<BlogPost> blogPostMatcher = passedBlogPost ->
            "Shane".equals(passedBlogPost.getAuthor());

    Mockito.when(blogRepository.checkIfBlogPostHasBeenSaved(Mockito.argThat(blogPostMatcher)))
            .thenReturn(true);
    BlogPostService service = new BlogPostService(blogRepository);

    // act
    boolean actual = service.checkIfBlogPostHasBeenSaved(post);

    // assert
    assertTrue(actual);
}

Verify a Stubbed Method Has Been Called

@Test
public void getAllBlogPosts_withBlogPost_verifiesThatRepositoryWasCalled() {
    // arrange
    List<BlogPost> expected = Arrays.asList(new BlogPost("Spring for humans"),
            new BlogPost("Mockito Cheatsheet"));
    BlogRepository blogRepository = Mockito.mock(BlogRepository.class);
    Mockito.when(blogRepository.getAllBlogPosts()).thenReturn(expected);
    BlogPostService service = new BlogPostService(blogRepository);

    // act
    service.getAllBlogPosts();

    // assert
    Mockito.verify(blogRepository).getAllBlogPosts();
}

Verify a Stubbed Method Hasn't Been Called

@Test
public void getBlogPostById_withBlogPost_verifiesThatCorrectMethodWasCalled() {
    // arrange
    BlogRepository blogRepository = Mockito.mock(BlogRepository.class);
    BlogPost expoected = new BlogPost("Spring for humans");
    Mockito.when(blogRepository.getBlogPostById(1)).thenReturn(Optional.of(expoected));
    BlogPostService service = new BlogPostService(blogRepository);

    // act
    service.getBlogPostById(1);

    // assert
    Mockito.verify(blogRepository, Mockito.never()).getBlogPostById(100);
    Mockito.verify(blogRepository).getBlogPostById(1);
}

Capturing Arguments

@Test
public void getBlogPostById_withBlogPost_verifiesCorrectArgumentPassed() {
    // arrange
    BlogRepository blogRepository = Mockito.mock(BlogRepository.class);
    int expected = 1;
    Mockito.when(blogRepository.getBlogPostById(expected))
            .thenReturn(Optional.of(new BlogPost("Spring for humans")));
    BlogPostService service = new BlogPostService(blogRepository);
    ArgumentCaptor<Integer> captor = ArgumentCaptor.forClass(Integer.class);

    // act
    service.getBlogPostById(expected);
    Mockito.verify(blogRepository).getBlogPostById(captor.capture());
    int actual = captor.getValue();

    // assert
    assertEquals(expected, actual);
}

Conclusion

Hopefully the snippets laid out here will help you quickly set up your mockito tests. The full examples can be seen here on GitHub.

Happy Coding!

Spring For Humans

Shane Jennings — Sun, 26 Apr 2020 15:51:48 +0000

I remember the first week of my first ever development job, my team lead came to me and said "We use Spring and Hibernate here, you'll need to learn them". I looked at him blankly, but thought "Hey, I'm sure if I Google this I'll figure it out in no time, I'm a University student, I'm smart right?".

I tried googling for Spring and came across blog post after blog post - I saw lots of mentions of words like "Dependency Injection" and "Inversion of Control". I also saw reams of XML which made no sense to me.

What I realised was that the people who were writing these blog posts were undeniably smart - but could not explain concepts to people who didn't understand even the basics of the concept.

So this is my goal for this blog post: explain Spring to you like a human. I want to answer the questions I had when I first started looking at Spring.

What is Spring Anyway?

When people say "Spring" (assuming they are talking about software development and not DIY or Slinkys) they usually mean the Spring Framework. Over the past few years it's possible that people may now refer more to Spring Boot when they say "Spring", but let's start with the basics first.

The Spring framework "provides a comprehensive programming and configuration model for modern Java-based enterprise applications". So what does that mean? Basically, the Spring framework allows you to create an application without having to worry about how it's all tied together. The Spring framework is comprised of a few different parts:

Core
Testing
Data Access
Web Framework
Integration
Language Support

In this post, we'll focus on Core, and specifically dependency management.

Spring Core

Dependency Injection

A big part of Spring Core is this idea of dependency injection. Dependency injection is a design pattern that makes a class independent of its dependencies (its variables).

Let's look at an example:

public class BlogPostService {

    private PostgresDatabaseRepository databaseRepository 
        = new PostgresDatabaseRepository();

    public BlogPost getBlogPostById(String id) {
        return databaseRepository.getBlogPostById(id);
    }
}

public class PostgresDatabaseRepository {

    public PostgresDatabaseRepository() {
        // connect to our database here
    }

    public BlogPost getBlogPostById(String id) {
        // We would go to our database here and
        // get our blog post
        BlogPost blogPost = new BlogPost();
        blogPost.setId(id);
        blogPost.setContent("my blog post content");
        blogPost.setName("Spring for Humans");
        return blogPost;
    }
}

public class BlogPost {
    private String id;
    private String name;
    private String content;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getContent() {
        return content;
    }

    public void setContent(String content) {
        this.content = content;
    }
}

Here we have a class BlogPostService which gets and returns a blog post by id from the database, by using DatabaseService. We have a few problems here:

We can't unit test BlogPostService independently of PostgresDatabaseRepository
BlogPostService is coupled to PostgresDatabaseRepository - what if we wanted to use MySQLDatabaseRepository instead? We would need to change our implementation of BlogPostService, which could then break our implementation and our test. You might think this is fine for one class, but what if we have 50 service classes that all need to change to use MySqlDatabaseRepository?

In fact, our implementation here breaks the "D" in the SOLID principles, where D is the Dependency inversion principle. This states that one should "depend upon abstractions, not concretions". This sounds fancy, but all it means is that our BlogPostService shouldn't depend on the class PostgresDatabaseRepository, which is a "concretion".

So what can we do to get around these problems? Dependency Injection to the rescue!

public class BlogPostService {

    private DatabaseRepository databaseRepository;

    // here BlogPostService's dependency is injected in
    public BlogPostService(DatabaseRepository databaseRepository) {
        this.databaseRepository = databaseRepository;
    }

    public BlogPost getBlogPostById(String id) {
        return databaseRepository.getBlogPostById(id);
    }
}

public class PostgresDatabaseRepository implements DatabaseRepository {

    public PostgresDatabaseRepository() {
        // connect to our database here
    }

    public BlogPost getBlogPostById(String id) {
        // We would go to our database here and
        // get our blog post
        BlogPost blogPost = new BlogPost();
        blogPost.setId(id);
        blogPost.setContent("my blog post content");
        blogPost.setName("Spring for Humans");
        return blogPost;
    }
}

public interface DatabaseRepository {
    BlogPost getBlogPostById(String id);
}

Now we can easily unit test our BlogPostService like so:

public class BlogPostServiceTest {

    @Test
    public void getBlogPostById_withValidId_returnsBlogPost() {
        // arrange
        BlogPost expected = new BlogPost();
        String id = "123";
        expected.setId(id);
        BlogPostService service = new BlogPostService(
            new MockDatabaseRepository(expected));

        // act
        BlogPost actual = service.getBlogPostById(id);

        // assert
        Assert.assertEquals(expected, actual);
    }

    private class MockDatabaseRepository implements DatabaseRepository {

        private BlogPost expected;

        public MockDatabaseRepository(BlogPost expected) {
            super();
            this.expected = expected;
        }

        public BlogPost getBlogPostById(String id) {
            if(id.equals(expected.getId())) return expected;
            return null;
        }
    }
}

Also, if we have a class MySqlDatabaseRepository which also implements DatabaseRepository then we can pass that class instead to the BlogService constructor. Success!

That's great and all, but if BlogPostService is no longer responsible for managing it's dependencies, then who is?

Well, we could always manage our dependencies ourselves like so:

public class Main {

    public static void main(String[] args) {
        BlogPostService blogPostService = DependencyContainer.getBlogPostService();
        blogPostService.getBlogPostById("123");
    }
}

public class DependencyContainer {

    public static BlogPostService getBlogPostService() {
        return new BlogPostService(getDatabaseRepository());
    }

    public static DatabaseRepository getDatabaseRepository() {
        return new PostgresDatabaseRepository();
    }
}

This works, however, it can get unwieldy. Especially if we have many dependencies and we want our dependencies to be Singletons (which we will do for classes like PostgresDatabaseRepository, we only need 1).

So what can we do?

Spring to the Rescue

This is where the beauty of Spring comes in. Spring Core contains what is called the "Inversion of Control container" (more on the term "Inversion of Control" later). This container is represented by the org.springframework.context.ApplicationContext interface and is responsible for instantiating, configuring, and assembling our dependencies. It does this by reading metadata in the form of XML, Java annotations, or Java code. I would strongly recommend not using XML - it's outdated and much harder to read and debug.

So let's see how we can do the above with Spring.

First, let's add a pom.xml to include our Spring maven dependencies:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.stacktobasics</groupId>
    <artifactId>Spring-For-Humans</artifactId>
    <version>1.0-SNAPSHOT</version>
    <dependencies>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-core</artifactId>
            <version>4.3.10.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
            <version>4.3.10.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.hk2.external</groupId>
            <artifactId>javax.inject</artifactId>
            <version>2.5.0-b32</version>
        </dependency>
    </dependencies>
</project>

Next, lets add our Spring configuration class:

import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import springforhumans.withspring.SpringConfiguration;

public class Main {

    public static void main(String[] args) {
        // Set up Spring
        AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext();
        context.register(SpringConfiguration.class);
        context.refresh();

        BlogPostService blogPostService = context.getBean(BlogPostService.class);
        BlogPost blogpost = blogPostService.getBlogPostById("123");
        System.out.println(String.format("Blog post id='%s' name='%s' content='%s'", blogpost.getId(), blogpost.getName(), blogpost.getContent()));
    }
}

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import springforhumans.BlogPostService;
import springforhumans.DatabaseRepository;
import springforhumans.PostgresDatabaseRepository;

@Configuration
public class SpringConfiguration {

    @Bean
    public BlogPostService blogPostService(DatabaseRepository databaseRepository) {
        return new BlogPostService(databaseRepository);
    }

    @Bean
    public DatabaseRepository databaseRepository() {
        return new PostgresDatabaseRepository();
    }
}

public class BlogPostService {

    private DatabaseRepository databaseRepository;

    public BlogPostService(DatabaseRepository databaseRepository) {

        this.databaseRepository = databaseRepository;
    }

    public BlogPost getBlogPostById(String id) {
        return databaseRepository.getBlogPostById(id);
    }
}

/ **All other classes have same implementation** /

Let's unpack what's going on here:

We create a configuration class called SpringConfiguration which has the annotation @Configuration. This tells Spring that it is a dependency configuration class.
The SpringConfiguration sets up our beans - don't be worried about this term, bean just means a standard Java object.
Spring automatically knows which DatabaseRepository implementation to pass into the blogPostService() method, as we already defined our DatabaseRepository bean - Spring intelligently figures out the dependency wiring order.
we can get our beans by doing context.getBean(MyClass.class)
Spring creates each of these beans as singletons by default, meaning that each time you call context.getBean(MyClass.class) you'll get the same object of that class.

Hmmm, this doesn't look like much less work than our implementation above. Let's look at another way of doing this with Spring:

package springforhumans;

import org.springframework.context.annotation.AnnotationConfigApplicationContext;

public class Main {

    public static void main(String[] args) {
        // Set up Spring
        AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext("springforhumans");

        BlogPostService blogPostService = context.getBean(BlogPostService.class);
        BlogPost blogpost = blogPostService.getBlogPostById("123");
        System.out.println(String.format("Blog post id='%s' name='%s' content='%s'", blogpost.getId(), blogpost.getName(), blogpost.getContent()));
    }
}

@Component
public class BlogPostService {

    private DatabaseRepository databaseRepository;

    public BlogPostService(DatabaseRepository databaseRepository) {

        this.databaseRepository = databaseRepository;
    }

    public BlogPost getBlogPostById(String id) {
        return databaseRepository.getBlogPostById(id);
    }
}

@Component
public class PostgresDatabaseRepository implements DatabaseRepository {

    public PostgresDatabaseRepository() {
        // connect to our database here
    }

    public BlogPost getBlogPostById(String id) {
        // We would go to our database here and
        // get our blog post
        BlogPost blogPost = new BlogPost();
        blogPost.setId(id);
        blogPost.setContent("my blog post content");
        blogPost.setName("Spring for Humans");
        return blogPost;
    }
}

This looks better! Let's again unpack what's happening here:

We got rid of the SpringConfiguration class.
We instead specified our base package to the AnnotationConfigApplicationContext constructor. This scans for any class which contains the @Component annotation.
We added @Component to our classes.

Now in this example, it doesn't look like we've saved ourselves much effort by using Spring - however, the benefit is more obvious when we have more dependencies to manage - trying to manage our dependencies can quickly become a mess.

It is important to note the price that comes with using @Component. Our classes now have a dependency on Spring, as they import the @Component annotation, which they did not have to do when we were using the SpringConfiguration class. Realistically you're not likely to switch to a different dependency Injection container in the same project, but it's important to be aware of nonetheless.

It is also important to note that using base package scanning with @Component and using SpringConfiguration are not mutually exclusive. We can use both in the same project and I regularly do. SpringConfiguration is great if your bean requires a more complex setup, which often happens when setting up Database connectors.

When should I not use dependency injection?

I'm sure you'll agree that dependency injection looks great - let's use it everywhere! Don't do that. Like everything dependency injection has its place. Most often dependency injection is suitable for classes that are:

stateless
will be instantiated as a singleton (but they don't have to be, see here for details.)
Will be reused
services, database connectors, and infrastructure connectors (e.g. REST Controllers).

A bad candidate for dependency injection is our BlogPost class - objects of BlogPost will contain a unique state for each blog post and will not be singletons. It would not make sense to try to inject BlogPosts into other classes.

Dependency Injection Vs Inversion of Control

You might have heard of the term "dependency Injection" (DI) and "inversion of Control" (IoC) floating around. Often people say that they are the same thing, however, I think of them as different concepts.

In terms of Spring:

IoC - the container which inhibits the ability to use dependency injection. For me, the ApplicationContext is the IoC Container, which follows the pattern of the framework calling the application, rather than the application calling the framework (so the application is not dependent on a framework).
DI - The act of a class allowing its dependencies to be injected, rather than having the responsibility to instantiate its dependencies.

Why would I need Spring?

Hopefully, I have already answered this question!

Spring is great for managing dependencies as already discussed. However, the Spring framework contains many more features than just dependency injection. Remember the list of Spring Framework features I listed in What is Spring anyway?. The beauty of Spring is that the more you use its features the more benefits you get as, unsurprisingly, the creators of Spring have made sure that their projects play together nicely.

And it's not just Spring Core. Spring has a multitude of different projects. Some of these include:

Spring Boot: Easily create standalone Spring applications that are wired together automatically, by using the @SpringBootApplication annotation. Often used to create web applications but can also be used as stand-alone publishers, subscribers, and more.
Spring Data: Easily create repository interfaces for your ORM objects, including automatically deriving queries from your method names.
Spring Cloud: Provides tools for building cloud-ready applications.

This is just to name a few, more can be seen on the Spring website.

What are some alternatives to Spring?

Spring is a big topic, but let's list some alternatives to the Spring Framework and Spring Boot.

Alternatives to Spring Framework (DI):

Guice
Dagger

Alternatives to Spring Boot:

I think the one to look out for is Micronaut - It's getting a lot of traction in the Java community due to its compatibility with GraalVM and its lack of use of Reflection.

What you've learned

Well done for sticking to the end!

Spring can appear overwhelming and confusing, and indeed it is a big subject. Much bigger than this blog post has room for. However, like with anything, start small and build your knowledge over time.

I would recommend getting your head around using Spring for dependency injection by trying the examples laid out here in this blog yourself and then giving Spring Boot a go. Spring's guide on creating a RESTful web service is a good place to start.

Happy Coding!