DEV Community: Nicolas Torres

How to use DEV.to as a CMS to make a blog with NextJS

Nicolas Torres — Thu, 11 Jan 2024 10:08:29 +0000

If you need a blog on your website, you can use DEV.to as a CMS to save a lot of time thanks to their API. Your editorial line should be tech-related though, as your articles will be published on both your website and DEV.

As PlaceKit is a tech startup, it is essential to host a blog to increase our search engines presence. Paying for a hosted CMS was not an option as we are self-funded, and self-hosting an open-source CMS would have cost us some setup and maintenance time that we'd rather allocate to more important matters. So building on top of DEV felt like a reasonable option, as we could also leverage its visibility.

We'll go through all the steps to create a blog in NextJS sourcing its articles from DEV API:

Preparing NextJS
Preparing methods
Listing articles
Article page
Updating canonical for SEO
Bonus: handling pagination

👉 See also: DEV API reference.

Don't let the article length or the number of code snippets intimidate you, I had to cover all possible implementation strategies with NextJS, using /pages or /app, SSG or SSR... 🥲

1. Preparing NextJS

Dependencies

DEV API serves articles in markdown, so we will be using Marked to parse it to HTML, and Prism for code syntax highlighting. Add them to your project:

npm install --save marked prismjs

Environment variables

Let's start with environment variables. We will need these:

# Enable articles search engines indexing
# do NOT set in development, uncomment when in production
# NEXT_PUBLIC_INDEXING=true

# Your website base URL
NEXT_PUBLIC_BASE_URL=http://localhost:3000
# NEXT_PUBLIC_BASE_URL=https://example.com

# DEV settings
DEVTO_API_KEY=<your-api-key>
DEVTO_USERNAME=<your-username>

Create an API key in your Account settings > Extensions > DEV Community API Keys.
Replace <your-api-key> in the env vars with the generated API key.
Replace <your-username> in the env vars with your DEV profile username or organization username.

⚠️ Important note: make sure the build environment for production has all these environment variables set. Sometimes we forget to add them when we're building in a CI environment like Google Cloudbuild for example.

Pages structure with `/pages` router

.
├── ...
├── pages
│   ├── ...
│   ├── blog                   # Blog route
│   │   ├── index.jsx          # Blog home page (articles list)
│   │   └── articles           # Blog articles
│   │       └── [slug].jsx     # Blog article dynamic route page
│   └── ...
└── ...

Pages structure with `/app` router

.
├── ...
├── app
│   ├── ...
│   ├── blog                   # Blog route
│   │   ├── page.jsx           # Blog home page (articles list)
│   │   └── articles           # Blog articles
│   │       └── [slug]         # Blog article dynamic route
│   │           └── page.jsx   # Blog article page
│   └── ...
└── ...

2. Preparing Methods

We're writing all our DEV API calls as reusable functions that will be common to all NextJS strategies, and prevent some bulk in your component pages.

I personnally like to put them in a utils folder, but you're free to place them wherever you want to. For the convenience of this tutorial, I'll place those helpers in ./utils/blog-methods.js and import them using the default absolute path alias @/utils/blog-methods.js.

// /utils/blog-methods.js
import { marked } from 'marked';

// fetch all articles, looping through the paginated DEV API route
export const getAllArticles = async (fetchOptions) => {
  let articles = [];

  // URL for user articles
  const url = new URL(`https://dev.to/api/articles/${process.env.DEVTO_USERNAME}/all`);
  // URL for organization articles
  // const url = new URL(`https://dev.to/api/organizations/${process.env.DEVTO_USERNAME}/articles`);

  // set default query parameters
  url.searchParams.set('page', 1);
  url.searchParams.set('per_page', 10);
  do {
    // fetch current page articles
    const res = await fetch(url, {
      ...fetchOptions,
      headers: {
        ...fetchOptions?.headers,
        'api-key': process.env.DEVTO_API_KEY,
      },
    });

    // stop looping on empty response
    if (!res.ok) {
      throw Error('Failed to fetch articles');
    }
    const data = await res.json();
    if (!data?.length) {
      break;
    }

    // store fetched articles
    articles = articles.concat(data);

    // increment page query parameter
    url.searchParams.set('page', +url.searchParams.get('page') + 1);
  } while (true);

  // return all articles
  return articles;
};

// fetch article by slug
export async function getArticleBySlug(slug, fetchOptions) {
  const res = await fetch(`https://dev.to/api/articles/${process.env.DEVTO_USERNAME}/${slug}`, {
    ...fetchOptions,
    headers: {
      ...fetchOptions?.headers,
      'api-key': process.env.DEVTO_API_KEY,
    },
  });
  if (!res.ok) {
    return null;
  }
  const data = await res.json();

  // convert markdown to HTML
  data.html = marked.parse(data.body_markdown);
  return data;
}

3. Listing articles

In our first version, we are focusing on retrieving and showing all articles on the same page. Depending on your NextJS strategy, you will need to call getAllArticles accordingly:

`/pages` router

// /pages/blog/index.jsx
import { getAllArticles } from '@/utils/blog-methods.js';

export default function BlogHome({ articles }) {
  // your page template
  return (
    /* ... */
  );
}

// To generate at build time (SSG), use `getStaticProps`
// To render at request time (SSR), use `getServerSideProps` instead
export async function getStaticProps() {
  const articles = await getAllArticles();
  return {
    props: {
      articles,
    },
  };
}

SSR with getServerSideProps in /pages router is not recommended because it won't cache the HTTP call, and therefore call DEV API each time a visitor opens the page.

`/app` router

// /app/blog/page.jsx
import { getAllArticles } from '@/utils/blog-methods.js';

export default async function Page() {
  // fetch data
  const articles = await getAllArticles({
    cache: 'force-cache', // set 'no-store' to always fetch at request time
    next: { revalidate: 24 * 60 * 60 * 1000 }, // revalidate cache every day
  });

  // your page template
  return (
    /* ... */
  );
}

Link articles

You now have access to the articles arrays within your page component, with a lot of details to display their title, description, tags, date and some other metadata. See the full JSON response to get the exhaustive list of available properties.

To link an article from the blog home, simply use href={`/blog/articles/${article.slug}`}:

// /pages/blog/index.jsx OR /app/blog/page.jsx
import Link from 'next/link';
// ...
export default function BlogHome({ articles }) {
  // ...
  return (
    <ul>
      {articles?.map((article) => {
        return (
          <li key={article.id.toString()}>
            <Link href={`/blog/articles/${article.slug}`}>{article.title}</Link>
          </li>
        );
      })}
    </ul>
  );
}
// ...

4. Article page

Again, depending on the NextJS strategy, we'll load the article and handle 404s in different ways. The main difference with the blog home page here is that in SSG/ISG mode, we need to tell NextJS all the pages that it has to generate.

`/pages` router, SSR mode

// /pages/blog/article/[slug].jsx
import { getArticleBySlug } from '@/utils/blog-methods.js';

// Article page component
export default function BlogArticle({ article }) {
  return <div dangerouslySetInnerHTML={{ __html: article.html }} />;
}

// Fetch article data at request time and inject as page props
export async function getServerSideProps({ params }) {
  const article = await getArticleBySlug(params.slug);
  return !article
    ? { notFound: true } // show 404 if fetching article fails
    : {
        props: {
          article,
        },
      };
}

`/pages` router, SSG mode

// /pages/blog/article/[slug].jsx
import { getAllArticles, getArticleBySlug } from '@/utils/blog-methods.js';

// Article page component
export default function BlogArticle({ article }) {
  return <div dangerouslySetInnerHTML={{ __html: article.html }} />;
}

// Pregenerate all article pages in SSG
export async function getStaticPaths() {
  const articles = await getAllArticles();
  const paths = articles.map(({ slug }) => ({ params: { slug } }));
  return {
    paths,
    fallback: false, // show 404 if not on the articles list
  };
}

// Fetch article data at build time and inject as page props
export async function getStaticProps({ params }) {
  const article = await getArticleBySlug(params.slug);
  if (article === null) {
    throw Error(`Failed to fetch article ${params.slug}`);
  }
  return {
    props: {
      article,
    },
  };
}

`/app` router

// /app/blog/article/[slug]/page.jsx

import { notFound } from 'next/navigation';
import { getArticleBySlug, getAllArticles } from '@/utils/blog-methods.js';

const cacheParams = {
  cache: 'force-cache', // set 'no-store' to always fetch at request time
  next: { revalidate: 24 * 60 * 60 * 1000 }, // revalidate cache every day
};

// Article page component
export default async function Page({ params }) {
  // Fetch article data at request time
  const article = await getArticleBySlug(params.slug, { cache: 'no-store' });

  if (!article) {
    notFound(); // show 404 if fetching article fails
  }

  return <div dangerouslySetInnerHTML={{ __html: article.html }} />;
}

// (Optional) Pregenerate all article pages at build time
export async function generateStaticParams() {
  const articles = await getAllArticles(cacheParams);
  return articles.map(({ slug }) => ({ slug }));
}

Adding anchors and code snippets syntax highlighting

Finally, let's update our utils/blog-methods.js file to configure Prism and Marked to add heading anchors and code syntax highlighting:

// /utils/blog-methods.js
import { marked } from 'marked';
import Prism from 'prismjs';

// load only languages you use
import 'prismjs/components/prism-bash';
import 'prismjs/components/prism-css';
import 'prismjs/components/prism-javascript';
import 'prismjs/components/prism-jsx';
import 'prismjs/components/prism-json';
import 'prismjs/components/prism-markup-templating';
import 'prismjs/components/prism-markdown';

// we handle Prism highlighting manually
Prism.manual = true;

// marked options
marked.use({
  breaks: true,
  gfm: true,
  extensions: [
    {
      // syntax highlighting on `<code>`
      // defaults to `plain` if the language isn't imported
      name: 'code',
      renderer({ lang, text }) {
        const output =
          lang in Prism.languages
            ? Prism.highlight(text.trim(), Prism.languages[lang], lang)
            : text.trim();
        const syntax = lang in Prism.languages ? lang : 'plain';
        return `<pre><code class="language-${syntax}">${output}</code></pre>`;
      },
    },
    {
      // adding heading anchors
      // same ID syntax as DEV so we don't need to rewrite links href
      name: 'heading',
      renderer({ text, depth }) {
        const id = text
          .toLocaleLowerCase()
          .replace(/[^\w\s]/g, '')
          .trim()
          .replace(/\s/g, '-');
        return `<h${depth}><a name="${id}" href="#${id}"></a>${text}</h${depth}>`;
      },
    },
  ],
});

// ...

And then import the CSS file for your prefered Prism theme in your article page:

// /pages/blog/article/[slug].jsx or /app/blog/article/[slug]/page.jsx
import 'prismjs/themes/prism.css'; // default theme

5. Updating canonical for SEO

Google and other search engine don't like much duplicate content, and will penalize your referencing if you simply post the same article on both DEV and your website.

To avoid this, we make use of the canonical URL, telling the search engines that the original post is in our website, and that DEV is a legitimate duplicate. So your website will rank articles in priority.

Make sure NEXT_PUBLIC_INDEXING is not set in development, and set to true in production, because we will update the DEV article canonical value with the public article URL. We create a new helper updateArticleCanonical and call it from the getArticleBySlug helper like so:

// /utils/blog-methods.js
// ...

// fetch article by slug
export async function getArticleBySlug(slug, fetchOptions) {
  /* ... */

  // update canonical on DEV
  await updateArticleCanonical(data);
  return data;
}

// update canonical URL when it goes public only if it isn't already set
async function updateArticleCanonical(article) {
  if (
    process.env.NEXT_PUBLIC_INDEXING &&
    !article.canonical_url.startsWith(process.env.NEXT_PUBLIC_BASE_URL)
  ) {
    const canonical = new URL(`/blog/articles/${article.slug}`, process.env.NEXT_PUBLIC_BASE_URL);
    await fetch(`https://dev.to/api/articles/${article.id}`, {
      method: 'PUT',
      body: JSON.stringify({
        article: {
          canonical_url: canonical.href,
          tags: article.tags, // for some reason, if not set, tags will be erased
        },
      }),
    });
  }
}

On a side note, we use next-sitemap to automatically generate a sitemap.xml, and our blog generated pages are processed without any specific config.

6. Bonus: handling pagination

Handling pagination in the blog home page is a matter of playing with NextJS router optional catch-all feature, like we did for prerendering individual articles at build time.

We could also use simple dynamic routes like /pages/blog/pages/[page].jsx and redirect /blog to /blog/page/1 in next.config.js, but I find it more elegant to use /blog and /blog/page/2, and the examples will be more exhaustive that way, for you to be able to choose what you prefer.

So let's add a redirect entry in our next.config.js for /blog/page/1 to be redirected permanently to /blog:

const nextConfig = {
  // ...
  async redirects() {
    return [
      {
        source: '/blog/page/1',
        destination: '/blog',
        permanent: true,
      },
    ];
  },
};

module.exports = nextConfig;

We still then need to load all articles with our getAllArticles helper from DEV to tell Next how many pages it will have to generate.

Folder structure

Let's first rename /pages/blog/index.jsx to /pages/blog/[[...page]].jsx, or /app/blog/page.jsx to /app/blog/[[...page]]/page.jsx to be able to catch both /blog, /blog/page/2, etc.

Careful that now, with that optional paramater, /blog/invalid/path will also be caught, so we need to control what paths we allow and show a 404 for the others.

Adding pagination data to articles list

We're wrapping getAllArticles to add some pagination data that will be common to all Next strategies:

// /utils/blog-methods.js
// ...

export const articlesPerPage = 10;

// fetch all articles with pagination
// careful, page query param is a 1-based integer for URL display
export async function getArticlesAtPage(params, fetchOptions) {
  const allArticles = await getAllArticles(fetchOptions);
  const pageParam = Number(params.page?.[1] || '1'); // 1-based
  const currentPage = Math.max(0, pageParam - 1); // 0-based for computations
  const start = currentPage * articlesPerPage;
  const end = start + articlesPerPage;
  return {
    articles: allArticles.slice(start, end),
    currentPage: currentPage + 1, // 1-based for display
    nbPages: Math.ceil(allArticles.length / articlesPerPage),
    perPage: articlesPerPage,
    nbArticles: allArticles.length,
  };
}

`/pages` router, SSR mode

// /pages/blog/index.jsx

import { getArticlesAtPage } from '@/utils/blog-methods.js';

//...

export async function getServerSideProps({ params }) {
  const pageProps = await getArticlesAtPage(params);
  return !pageProps.articles.length
    ? { notFound: true }
    : {
        props: pageProps, // { articles, currentPage, nbPages, ... }
      };
}

`/pages` router, SSG mode

// /pages/blog/index.jsx

import { getAllArticles, getArticlesAtPage, articlesPerPage } from '@/utils/blog-methods.js';

// ...

export async function getStaticPaths() {
  // fetching all articles here to compute the total number of pages
  const allArticles = await getAllArticles();
  const nbPages = Math.ceil(allArticles.length / articlesPerPage);
  const pages = Array.from(Array(nbPages).keys()); // [0,1,...,N-1]
  const paths = pages.map((n) => ({
    params: {
      page:
        n === 0
          ? null // `/blog`
          : ['page', `${n + 1}`], // pages [2,3,...,N]
    },
  }));
  return {
    paths,
    fallback: false, // show 404 if not on the pages list
  };
}

export async function getStaticProps({ params }) {
  const pageProps = await getArticlesAtPage(params);
  return {
    props: pageProps, // { articles, currentPage, nbPages, ... }
  };
}

`/app` router

// /app/blog/page.jsx
import { notFound } from 'next/navigation';
import { getAllArticles, getArticlesAtPage, articlesPerPage } from '@/utils/blog-methods.js';

// ...

const cacheParams = {
  cache: 'force-cache', // set 'no-store' to always fetch at request time
  next: { revalidate: 24 * 60 * 60 * 1000 }, // revalidate cache every day
};

export default async function Page({ params }) {
  const pageProps = await getArticlesAtPage(params, cacheParams);

  if (!pageProps.articles?.length) {
    notFound();
  }

  // your page template
  return {
    /* ... */
  };
}

// (Optional) Pregenerate all article pages at build time
export async function generateStaticParams() {
  const articles = await getAllArticles(cacheParams);
  const nbPages = Math.ceil(articles.length / articlesPerPage);
  const pages = Array.from(Array(nbPages).keys()); // [0,1,...,N-1]
  return pages.map((n) => ({
    page:
      n === 0
        ? null // `/blog`
        : ['page', `${n + 1}`], // pages [2,3,...,N]
  }));
}

Check out our blog at placekit.io/blog for a live implementation example (and more articles 👀).

We hope this tutorial helped you setting up DEV as a CMS for your own NextJS website!

Introducing Live Patching

Nicolas Torres — Tue, 11 Jul 2023 04:51:43 +0000

Live Patching empowers you to add or fix locations on the fly with PlaceKit, and make them instantly available to your users.

Motivation

Geocoding is providing geographical coordinates corresponding to a location, and reverse geocoding means finding a location based on geographical coordinates. One common use-case of geocoding you may have seen is an address autocomplete field:

Providing geocoding services means keeping up to date a catalog of worldwide locations with their coordinates, and providing the API to browse through that immense amount of data. The catalog is an overly tedious task, which always requires some degree of automation and approximation. Thus no one, not even Google Maps, can pretend to have it 100% accurate.

When we were working on Algolia Places (before it got sunset), we noticed a critical pain point for our customers. Some use cases need a way to fix locations from a catalog, without delay. For example, an e-commerce website experiencing mis-shipments due to an invalid address will often reach out to its user to get the correct one, and need a way to fix it once and for all to prevent any more failed delivery for that person. And that would occur multiple times per week.

Even if we were somewhat reactive in fixing locations at Algolia Places, it had to go through a submission process via a form, manually reviewing internally whenever the product team has the time. It's very time-consuming and a bit hard to propagate fixes in a large automated pipeline processing. The same goes with our competitors like Google Places or Mapbox: at best, fixing a location is manual a process that could take days or weeks of back-and-forth validation.

At PlaceKit, we believe businesses know better, so we're introducing a differentiating feature: Live Patching, to let you add missing locations or fix existing ones without delay, providing the best experience possible for your users.

Instead of waiting for your favorite geocoding service employees to review and validate/reject your submission, PlaceKit enables you to build your own validation flow. You can either create fixes directly from the dashboard or API, or review and validate suggestions coming from your users. Once you approved the changes, the patches are instantly available to all your end users.

How does it work

We merge your published patches in the autocomplete suggestions at search time, only for your applications:

For missing locations, we append them to the search, and it goes through the same relevance algorithm as any other location from our catalog.
For fixes, we detect and substitute them from our original record, so there are no duplicates in the suggestions list.

Patches are tied to your PlaceKit account, so they're shared across all your applications, but not shared with other PlaceKit customers: only your users see your fixes.

Using the admin panel

Sign in to your PlaceKit account and navigate to the new "Patch" tab. You should see an empty list and an "Add patch" button. Click on it to access the form. For example, adding a new street named "New Street" to Tahoe City:

When adding a missing location, you can pre-fill the form with an existing location. For example, if you're adding a missing street, that could come in handy to fill in all the city information.

When fixing an existing location, it'll be required that you search for the existing location first. It'll fill the form for you, and then you only have to fix the wrong fields.

You can publish and unpublish it at any time. Published means your users will see it when they search for a location; unpublished means it's only showing on your Patch page on the admin panel.

Now you have it ready for your users in the autocomplete:

Using the API

We've updated our API Reference and the JS Client with the new endpoints.

⚠️ You need to use a private API key from any of your apps.

Snippets below showcase both cURL and JavaScript implementations. Before running the JS examples, instantiate PlaceKit Client:

// CommonJS syntax:
const placekit = require('@placekit/client-js');

// ES6 Modules syntax:
import placekit from '@placekit/client-js';

const pk = placekit('<your-api-key>');

Now let's make a tour of the feature:

Adding a missing location

const result = await pk.patch.create({
 type: 'street',
 name: 'New street',
 city: 'Tahoe City',
 county: 'Placer County',
 administrative: 'California',
 country: 'United States of America',
 countrycode: 'us',
 coordinates: '39.1689372,-120.1434787',
 zipcode: ['96154'],
 population: 2644
});

curl -l -X POST 'https://api.placekit.co/patch' \
 -H 'x-placekit-api-key: <your-api-key>' \
 -H 'Content-Type: application/json' \
 -d '{"record":{"type":"street","name":"New street","city":"Tahoe City","county":"Placer County","administrative":"California","country":"United States of America","countrycode":"us","coordinates":"39.1689372,-120.1434787","zipcode":["96154"],"population":2644}}'

Fixing an existing location

// Search for an existing record first
const { results } = await pk.search('Los Angeles', {
 types: ['city'],
 countries: ['us'],
});

// Fix the existing record
await pk.patch.create(
 { zipcode: ['96154'] },
 { status: 'approved' },
 results[0], // original record
);

curl -l -X PUT 'https://api.placekit.co/patch' \
 -H 'x-placekit-api-key: <your-api-key>' \
 -H 'Content-Type: application/json' \
 -d '{"update":{"zipcode":["96154"]},"origin":{/* original record */},"status":"approved"}'

Listing patch records

// Get all patches, paginated
const { results } = await pk.patch.list();

// Filter and paginate patches
const { results } = await pk.patch.list({
 query: 'new',
 countries: ['us'],
 types: ['street'],
 status: 'approved',
 maxResults: 10,
 offset: 0,
});

# Get all patches, paginated
curl -l -X POST 'https://api.placekit.co/patch/search' \
 -H 'x-placekit-api-key: <your-api-key>'

# Filter and paginate patches
curl -l -X POST 'https://api.placekit.co/patch/search' \
 -H 'x-placekit-api-key: <your-api-key>' \
 -H 'Content-Type: application/json' \
 -d '{"query":"new","countries":["us"],"status":"approved","maxResults":10,"offset":0}'

Retrieving a patch record

// Retrieve default language
const result = await pk.patch.get('<patch-id>');

// Retrieve translation
const result = await pk.patch.get('<patch-id>', 'fr');

# retrieve default language
curl -l -X GET 'https://api.placekit.co/patch/<patch-id>' \
 -H 'x-placekit-api-key: <your-api-key>'

# retrieve translation
curl -l -X GET 'https://api.placekit.co/patch/<patch-id>?language=fr' \
 -H 'x-placekit-api-key: <your-api-key>'

Updating a patch record

// Update and publish
const result = await pk.patch.update(
 '<patch-id>',
 { coordinates: '39.1689372,-120.1434787' },
 { status: 'approved' }
);

// Update translation
const result = await pk.patch.update(
 '<patch-id>',
 { name: 'Rue Nouvelle' }, 
 { language: 'fr' }
);

// Unpublish
const result = await pk.patch.update(
 '<patch-id>',
 undefined,
 { status: "pending" }
);

# Update and publish
curl -l -X PATCH 'https://api.placekit.co/patch/<patch-id>' \
 -H 'x-placekit-api-key: <your-api-key>' \
 -H 'Content-Type: application/json' \
 -d '{"update":{"coordinates":"39.1689372,-120.1434787"},"status":"approved"}'

# Update translation
curl -l -X PATCH 'https://api.placekit.co/patch/<patch-id>' \
 -H 'x-placekit-api-key: <your-api-key>' \
 -H 'Content-Type: application/json' \
 -d '{"update":{"name":"Rue Nouvelle"},"language":"fr"}'

# Unpublish
curl -l -X PATCH 'https://api.placekit.co/patch/<patch-id>' \
 -H 'x-placekit-api-key: <your-api-key>' \
 -H 'Content-Type: application/json' \
 -d '{"status":"pending"}'

Delete a patch record

// Delete translation
await pk.patch.deleteLang('<patch-id>', 'fr');

// Delete patch record
await pk.patch.delete('<patch-id>');

# Delete translation
curl -l -X DELETE 'https://api.placekit.co/patch/<patch-id>/language/fr'

# Delete patch
curl -l -X DELETE 'https://api.placekit.co/patch/<patch-id>'

It's mostly just a developer feature for now, but stay tuned, we'll add more to it soon. Nonetheless, we're excited to bring this unique feature to our service, and can't wait to see your implementations!

Making React-Leaflet work with NextJS

Nicolas Torres — Fri, 23 Jun 2023 08:55:10 +0000

I've ran into some issues implementing React Leaflet with NextJS for our admin panel at PlaceKit. So let's gather my findings into a single article, hoping it'll save you some time.

Because NextJS has an SSR (server-side rendering) layer, importing third-party front-end libraries sometimes results in headaches.

Most of the time, you just have to wrap your front-end component with next/dynamic to make it lazy load, making the SSR pass simply ignore it:

// MyComponent.jsx
import frontLib from '<your-front-end-library>';
const MyComponent = (props) => {
  // do something with frontLib
};
export default MyLazyComponent;


// MyPage.jsx
import dynamic from 'next/dynamic';

const MyComponent = dynamic(
  () => import('./MyComponent'),
  {
    ssr: false,
    loading: () => (<div>loading...</div>),
  }
);

const MyPage = (props) => (
  <MyComponent />
);

But in the case of React Leaflet, you may need to put in some more efforts.

Passing `ref`

If you simply assign a ref to lazy-loaded <MapContainer>, you'll get a unusable proxy reference coming from dynamic:

// Map.jsx
import dynamic from 'next/dynamic';
import { useEffect, useRef } from 'react';

const MapContainer = dynamic(
  () => import('react-leaflet').then((m) => m.MapContainer),
  { ssr: false }
);

const Map = (props) => {
  const mapRef = useRef(null);
  useEffect(
   () => console.log(mapRef.current), // { retry: fn, ... }
   [mapRef.current]
  );
  return (
    <MapContainer ref={mapRef} ?>
  );
};

export default Map;

The trick is a bit bulky, but you have to wrap it under another component and forward the ref as a standard property (mapRef here), and you lazy load that one instead:

// MapLazyComponents.jsx
import {
  MapContainer as LMapContainer,
} from 'react-leaflet';

export const MapContainer = ({ forwardedRef, ...props }) => (
  <LMapContainer {...props} ref={forwardedRef} />
);

// Map.jsx
import dynamic from 'next/dynamic';
import { forwardRef, useEffect, useRef } from 'react';

const LazyMapContainer = dynamic(
  () => import('./MapLazyComponents').then((m) => m.MapContainer),
  { ssr: false }
);

const MapContainer = forwardRef((props, ref) => (
  <LazyMapContainer {...props} forwardedRef={ref} />
));

const Map = (props) => {
  const mapRef = useRef(null);
  useEffect(
   () => console.log(mapRef.current), // this works!
   [mapRef.current]
  );
  return (
    <MapContainer ref={mapRef} />
  );
};

export default Map;

Organizing your components

As we'll be preparing a few other React Leaflet components in the following examples, let's reorganise this into 3 files:

Map.jsx: your final component or page showing the map.
MapComponents.jsx: components that will lazy-load the React Leaflet ones. These will be ready to import as-is.
MapLazyComponents.jsx: wrappers that forward ref or are using front-end specific features, to be lazy-loaded by MapComponents.jsx.

Let's also add <TileLayer> and <ZoomControl> as we won't need any specific changes apart from loading them with dynamic.

So at this point you get:

// MapLazyComponents.jsx
import {
  MapContainer as LMapContainer,
} from 'react-leaflet';

export const MapContainer = ({ forwardedRef, ...props }) => (
  <LMapContainer {...props} ref={forwardedRef} />
);


// MapComponents.jsx
import dynamic from 'next/dynamic';
import { forwardRef } from 'react';

export const LazyMapContainer = dynamic(
  () => import('./MapLazyComponents').then((m) => m.MapContainer),
  {
    ssr: false,
    loading: () => (<div style={{ height: '400px' }} />),
  }
);
export const MapContainer = forwardRef((props, ref) => (
  <LazyMapContainer {...props} forwardedRef={ref} />
));

// direct import from 'react-leaflet'
export const TileLayer = dynamic(
  () => import('react-leaflet').then((m) => m.TileLayer),
  { ssr: false }
);
export const ZoomControl = dynamic(
  () => import('react-leaflet').then((m) => m.ZoomControl),
  { ssr: false }
);


// Map.jsx
import { useEffect, useRef } from 'react';

// import and use components as usual
import { MapContainer, TileLayer, ZoomControl } from './MapComponents.jsx';

const Map = (props) => {
  const mapRef = useRef(null);
  return (
    <MapContainer
      ref={mapRef}
      touchZoom={false}
      zoomControl={false}
      style={{ height: '400px', zIndex: '0!important' }}
    >
      <TileLayer url="..." attribution="..." style={{ zIndex: '0!important' }} />
      <ZoomControl position="topright" style={{ zIndex: '10!important' }} />
    </MapContainer>
  );
};

export default Map;

Using custom Marker icons

Alright, now that we start having a Map, let's add a marker. But most of the time you'd want to use a custom icon with it.

Custom Marker icons need to use L.Icon() from leaflet itself, which is a library instantiating stuff in window, so it breaks SSR when importing in Next. But it can not be loaded with dynamic() or even with React.lazy() which are exclusive to lazy loading components.

So, let's wrap our <Marker> component in MapLazyComponents.jsx as it'll be depending on front-end exclusive features:

// MapLazyComponents.jsx
import { useEffect, useState } from 'react';
import {
  MapContainer as LMapContainer,
  Marker as LMarker,
} from 'react-leaflet';

// ...
export const Marker = ({ forwardedRef, icon: iconProps, ...props }) => {
  const [icon, setIcon] = useState();

  useEffect(
    () => {
      // loading 'leaflet' dynamically when the component mounts
      const loadIcon = async () => {
        const L = await import('leaflet');
        setIcon(L.icon(iconProps));
      }
      loadIcon();
    },
    [iconProps]
  );

  // waiting for icon to be loaded before rendering
  return (!!iconProps && !icon) ? null : (
    <LMarker
      {...props}
      icon={icon}
      ref={forwardedRef}
    />
  );
};

// MapComponents.jsx
// ...
const LazyMarker = dynamic(() => import('./MapLazyComponents').then((m) => m.Marker), { ssr: false });
export const Marker = forwardRef((props, ref) => (
  <LazyMarker {...props} forwardedRef={ref} />
));

// Map.jsx
// ...
import { MapContainer, TileLayer, ZoomControl, Marker } from './MapComponents.jsx';

import CustomIcon from '../public/custom-icon.svg';

const Map = (props) => {
  const mapRef = useRef(null);
  const markerRef = useRef(null);
  return (
    <MapContainer
      ref={mapRef}
      touchZoom={false}
      zoomControl={false}
      style={{ height: '400px', zIndex: '0!important' }}
    >
      <TileLayer url="..." attribution="..." style={{ zIndex: '0!important' }} />
      <ZoomControl position="topright" style={{ zIndex: '10!important' }} />
      <Marker
        ref={markerRef}
        icon={{
          iconUrl: CustomIcon.src,
          iconAnchor: [16,32],
          iconSize: [32,32]
        }}
        style={{ zIndex: '1!important' }}
      />
    </MapContainer>
  );
};
//...

Handling map events

For marker events, you can already pass the eventHandlers property and it'll work. But to handle map events, it can not be done on the <MapContainer> component, you need to use the useMapEvents() hook from React Leaflet in a child component.

Same here, we'll need to wrap it, and we'll do it within a custom <MapConsumer> element to simplify things:

// MapLazyComponents.jsx
//...
import { useMapEvents } from 'react-leaflet/hooks';
export const MapConsumer = ({ eventsHandler }) => {
  useMapEvents(eventsHandler);
  return null;
};

// MapComponents.jsx
//...
export const MapConsumer = dynamic(
  () => import('./MapLazyComponents').then((m) => m.MapConsumer),
  { ssr: false }
);

So in you Map.jsx file, you're now able to add <MapConsumer> in <MapContainer>:

// Map.jsx
//...
const Map = (props) => {
  const mapRef = useRef(null);
  const markerRef = useRef(null);

  const mapHandlers = useMemo(
    () => ({
      click(e) {
        // center view on the coordinates of the click
        // `this` is the Leaflet map object
        this.setView([e.latlng.lat, e.latlng.lng]);
      },
    }),
    []
  );

  return (
    <MapContainer
      ref={mapRef}
      touchZoom={false}
      zoomControl={false}
      style={{ height: '400px', zIndex: '0!important' }}
    >
      <TileLayer url="..." attribution="..." style={{ zIndex: '0!important' }} />
      <ZoomControl position="topright" style={{ zIndex: '10!important' }} />
      <MapConsumer
        eventsHandler={mapHandlers}
      />
      <Marker
        ref={markerRef}
        icon={{
          iconUrl: CustomIcon.src,
          iconAnchor: [16,32],
          iconSize: [32,32]
        }}
        style={{ zIndex: '1!important' }}
      />
    </MapContainer>
  );
};

A few states and CSS later, here's my result:

So we've seen how to:

lazy load components with next/dynamic,
make ref work with lazy-loaded components,
dynamically load leaflet to access its methods like L.Icon,
wrap react-leaflet custom hooks to handle events.

Adapting these tricks should cover most of your edge cases. I hope breaking down into these specific use-cases will help you work better with React Leaflet on NextJS!

And of course, if you need a reverse geocoding API to get coordinates from an address, have a look at PlaceKit.io :)!

Run a worker alongside Next.js server using a single command

Nicolas Torres — Fri, 16 Apr 2021 09:53:55 +0000

By default Next.js only has one entry point: the web server, sourcing /pages. But if you're building a real API, you may need other entry points to run scripts and/or run a worker to process background jobs.

You could just add a worker.js file and execute it with node /path/to/worker.js but you'll lose ES6 imports and therefore compatibility with your helpers. No point in duplicating the Next.js build stack, let's see how we can reuse it.

Next.js allows us to extend its Webpack config in next.config.js, we only need to specify our new entry points there. As stated in my previous article Build a full API with Next.js:

const path = require('path');

module.exports = {
  webpack: (config, { isServer }) => {
    if (isServer) {
      return {
        ...config,
        entry() {
          return config.entry().then((entry) => ({
            ...entry,
            // adding custom entry points
            worker: path.resolve(process.cwd(), 'src/worker.js'),
            run: path.resolve(process.cwd(), 'src/run.js'),
          }));
        }
      };
    }
  },
};

Pretty basic. But how do we run them? node ./src/worker.js won't work because it needs go to through Webpack. So we have to wait for the file to have compiled with next dev or next start commands. Once your app is built, the compiled file will be available at .next/server/worker.js so we can basically just run node .next/server/worker.js and now it'll work!

But that's a poor developer experience, as we have to wait for first compilation before we run our worker process in a second terminal. To run the worker alongside the server with a single command, I rely on:

npm-run-all to execute multiple commands in parallel,
wait-on to wait for the file to exist before running the worker,
nodemon to reload the worker on file change.

Here's how my package.json looks like:

{
  //...
  "scripts": {
    "dev:app": "next dev"
    "dev:worker": "wait-on .next/server/worker.js && dotenv -c -- nodemon .next/server/worker.js -w src/server -w src/shared"
    "dev": "npm-run-all -p dev:worker dev:app",
    "worker": "dotenv -c -- node .next/server/worker.js",
    "script": "dotenv -c -- node .next/server/run.js script",
    "job": "dotenv -c -- node .next/server/run.js job",
    //...
  }
}

A few notes here:

I'm only watching back-end utilities with nodemon (src/server and src/shared) so front-end changes don't unnecessarily reload the worker.
I use dotenv-cli to source .env files because Next.js won't inject them in custom entry points.
Running a script or a job is managed here by a single entry point run.js but you could have 2 separate files to handle this. As it's an on-off process, I don't feel the need to use wait-on nor nodemon.

Hope this helps!

Fix "too many connections" errors with database clients stacking in dev mode with Next.js

Nicolas Torres — Sun, 11 Apr 2021 16:03:14 +0000

Next.js constantly rebuilds your API routes in development mode, triggering each time the creation of new instances of databases and other third-party services your back-end relies on.

This is not a very well documented part, and I've spent many hours looking for a solution, both online and within the source code of Next.js, to finally find a very simple yet elegant one: using global.

Indeed the server itself does not reload, so global is preserved across incremental builds. So here's the helper I use to prevent the server from stacking Redis, PostgreSQL and other SDK clients:

/**
 * Register service.
 * @description Stores instances in `global` to prevent memory leaks in development.
 * @arg {string} name Service name.
 * @arg {function} initFn Function returning the service instance.
 * @return {*} Service instance.
 */
const registerService = (name, initFn) => {
  if (process.env.NODE_ENV === 'development') {
    if (!(name in global)) {
      global[name] = initFn();
    }
    return global[name];
  }
  return initFn();
};

Yep, that's it. To use it, simply call:

import knex from 'knex';

export const db = registerService('db', () => knex({
  client: 'pg',
  connection: process.env.DATABASE_URL,
}));

Now you won't have to restart Next.js server every 10 minutes to flush out residual clients.

Bypassing Shopify Admin REST API limitations with a custom JS client

Nicolas Torres — Mon, 25 Jan 2021 14:49:01 +0000

Quick article to give away my custom Shopify Admin REST API client (based on Axios), that integrates a retry strategy along with a pagination sugar to overcome the very restrictive rate limit policy of Shopify. Note that it also removes the need to wraps the data sent under the resource name like { customer: { data } }.

Usage

const shopifyAdmin = require('./shopifyAdmin');

const myShop = shopifyAdmin({
  shopName: 'your-shop-name',
  version: '2021-01',
  apiKey: '**********',
  password: '**********',
});

myShop('POST /customers', { email: 'you@example.com' });
myShop('GET /customers/search', { query: 'email:"you@example.com"' });
myShop('GET[ALL] /orders'); // paginate and return all
myShop('GET[500] /orders'); // paginate and return first 500

Consider the performance drag of pagination and retry (whenever the API returns a 429). The retry strategy adds a 1 second delay before each call. Better use this in async jobs (like with Bull) whenever possible.

Client source code

const axios = require('axios');
const parseHeaderLink = require('parse-link-header');

/**
 * Custom Axios client for Shopify Admin.
 *
 * Implements pagination, retry strategy, and call frequency control as a workaround for Shopify Admin API call limits.
 *
 * @arg {Object} credentials Shopify Admin API credentials.
 * @arg {string} credentials.shopName Shopify shop name
 * @arg {string} credentials.version Shopify Admin API version
 * @arg {string} credentials.apiKey Shopify Admin API key
 * @arg {string} credentials.password Shopify Admin API password
 * @return {Function} Axios request wrapper.
 */
export default ({ shopName, version, apiKey, password }) => {
  if (!shopName || !version || !apiKey || !password) {
    throw Error('Shopify Admin init: missing credentials.');
  }

  const baseURL = `https://${shopName}.myshopify.com/admin/api/${version}`;
  const token = Buffer.from(`${apiKey}:${password}`).toString('base64');
  const client = axios.create({
    baseURL,
    headers: {
      Authorization: `Basic ${token}`,
    },
  });

  /**
   * Retry if too many requests.
   * @arg {Function} fn Function to execute.
   * @return {Promise<Object[]>} Axios response Promise.
   */
  const retry = async (fn) => new Promise((resolve, reject) => {
    setTimeout(() => {
      fn().then(resolve).catch((err) => {
        if (err.message.includes('429')) {
          retry(fn).then(resolve).catch(reject);
        } else {
          reject(err);
        }
      });
    }, 1000);
  });

  /**
   * Returns concatenated results after running pagination.
   * @arg {Object} config Axios request parameters
   * @arg {number} limit Number of results to return ('undefined' returns all)
   * @return {Promise<Object[]>} Axios response Promise.
   */
  const paginate = async (config, limit) => {
    let output = [];
    do {
      const res = await retry(() => client.request(config));
      output = output.concat(Object.values(res.data)[0]);
      const links = parseHeaderLink(res.headers.link);
      config.params = links && links.next ? {
        limit: links.next.limit,
        page_info: links.next.page_info,
      } : undefined;
    } while (config.params && (!limit || output.length <= limit));
    return output.slice(0, limit);
  };

  /**
   * Axios request wrapper.
   * @arg {string} action Method + URL combined (e.g. "GET /orders").
   * @arg {Object} [data] Body data.
   * @arg {Object} [params] Query parameters.
   * @return {Promise} Axios response Promise.
   * @example
   * // Get products
   * shopifyAdmin('/products')
   * shopifyAdmin('GET /products')
   * @example
   * // Add customer
   * shopifyAdmin('POST /customers', { email: 'you@example.com' })
   * @example
   * // Search customers
   * shopifyAdmin('GET /customers/search', { query: 'email:"you@example.com"' })
   * @example
   * // Get all orders (uses pagination)
   * shopifyAdmin('GET[ALL] /orders')
   * @example
   * // Get 500 orders (uses pagination)
   * shopifyAdmin('GET[500] /orders')
   */
  return (action, data = {}, params = {}) => {
    if (!action) {
      throw Error('Shopify Admin client: missing request URL.');
    }

    const [method, url] = action.trim().split(/\s+/);
    const config = { url: `${url || method}.json` };
    const superGetRegexp = /get\[(all|[0-9]+)\]/i;
    config.method = (url ? method.replace(superGetRegexp, 'GET') : 'GET').toUpperCase();
    if (['DELETE', 'GET'].includes(config.method)) {
      // pass data as query string
      config.params = {
        ...data,
        ...params,
      };
    } else {
      // wrap data under resource (ex: `{customer: data}`)
      const resource = url.replace(/^\/?(?:[a-z_0-9]+\/)*([a-z_]+)s(?:\/[0-9]+)*$/, '$1');
      config.data = { [resource]: data };
      config.params = params;
    }
    const paginated = method.match(superGetRegexp);
    return paginated ?
      paginate(config, parseInt(paginated[1], 10) || undefined) :
      retry(() => client.request(config).then((res) => Object.values(res.data)[0]));
  };
};

Hope this helps!

Using Fuse.js with React to build an advanced search (with highlighting)

Nicolas Torres — Tue, 19 Jan 2021 18:27:46 +0000

Fuse.js is really a great library to build a fuzzy (typo-tolerant) search engine in seconds, as it has great performances as long as the dataset size remains decent, and a very small and clear API.

But I've run into many difficulties trying to make it work with React. I had started by creating the Fuse instance inside a FuseSearch input component, and then passing the results with a callback property. But once we type, the parent receive the results, refresh its DOM, and therefore refresh my FuseSearch component updating the instance and triggering a new search... causing an infinite loop caught by React max update limit ("Maximum update depth exceeded."). (Won't share the code here, way too verbose for a misdirection.)

Creating a `useFuse` custom hook

To avoid the update loop, the solution is to instantiate Fuse in the component that will display the results directly. Thanks to balazssagi/use-fuse for that! It wraps the Fuse instance into a custom hook, so the usage becomes:

// 😶 decent solution, but not yet there
import React, { useCallback, useState } from 'react';
import { useFuse } from 'features/useFuse';
const MySearch = (props) => {
  const [query, setQuery] = useState('');
  const hits = useFuse(props.list, query);
  const onSearch = useCallback(
    (e) => setQuery(e.target.value.trim()),
    []
  );
  return (
    <input
      name="search"
      type="search"
      placeholder="Search..."
      autoComplete="off"
      onKeyUp={onSearch}
      onChange={onSearch} // handles "clear search" click
    />
    <p>Results for "{query}":</p>
    <ol>
      {hits.map(hit => (
        <li key={hit.refIndex}>
          {hit.item.name}
        </li>
      ))}
    </ol>
  );
};

However a few things bother me with this hook, as Fuse has many more options. I want to be able to:

tweak options for each instance,
avoid declaring query state and onSearch handler every time,
debounce search on keypress,
return the full list of results when query is empty, because Fuse won't (it's understandably out of its scope).

So let's rewrite the hook to make all of that happen:

// features/useFuse.js
import Fuse from 'fuse.js';
import { useCallback, useMemo, useState } from 'react';
import { debounce } from 'throttle-debounce';

export const useFuse = (list, options) => {
  // defining our query state in there directly
  const [query, updateQuery] = useState('');

  // removing custom options from Fuse options object
  // NOTE: `limit` is actually a `fuse.search` option, but we merge all options for convenience
  const { limit, matchAllOnEmptyQuery, ...fuseOptions } = options;

  // let's memoize the fuse instance for performances
  const fuse = useMemo(
    () => new Fuse(list, fuseOptions),
    [list, fuseOptions]
  );

  // memoize results whenever the query or options change
  const hits = useMemo(
    // if query is empty and `matchAllOnEmptyQuery` is `true` then return all list
    // NOTE: we remap the results to match the return structure of `fuse.search()`
    () => !query && matchAllOnEmptyQuery
      ? fuse.getIndex().docs.slice(0, limit).map((item, refIndex) => ({ item, refIndex }))
      : fuse.search(query, { limit }),
    [fuse, limit, matchAllOnEmptyQuery, query]
  );

  // debounce updateQuery and rename it `setQuery` so it's transparent
  const setQuery = useCallback(
    debounce(100, updateQuery),
    []
  );

  // pass a handling helper to speed up implementation
  const onSearch = useCallback(
    (e) => setQuery(e.target.value.trim()),
    [setQuery]
  );

  // still returning `setQuery` for custom handler implementations
  return {
    hits,
    onSearch,
    query,
    setQuery,
  };
};

Okay so now our example becomes way less verbose and we can set different options per instance:

// 🎉 Here we are!
import React from 'react';
import { useFuse } from 'features/useFuse';
const MySearch = (props) => {
  const { hits, query, onSearch } = useFuse(props.list, {
    keys: ['name'],
    matchAllOnEmptyQuery: true,
  });
  return (
    <input
      name="search"
      type="search"
      placeholder="Search..."
      autoComplete="off"
      onKeyUp={onSearch}
      onChange={onSearch} // handles "clear search" click
    />
    <p>Results for "{query}":</p>
    <ol>
      {hits.map(hit => (
        <li key={hit.refIndex}>
          {hit.item.name}
        </li>
      ))}
    </ol>
  );
};

We still have access to setQuery if we have to use a custom component hijacking the key events, from a UI kit for example.

Highlighting results

When turning on the option includeMatches, Fuse returns a matches object alongside each item:

[
  {
    item: {/* ... */},
    matches: {
      indices: [[1,1], [3,5]],
      key: 'path.to.key',
      value: "The value of item[path][to][key]",
    },
    refIndex: 0,
  },
  // ...
]

But again, I couldn't find any satisfying highlighting component for Fuse, so I've built my own, FuseHighlight:

// components/FuseHighlight.jsx
import React from 'react';

// Finds `obj[path][to][key]` from `path.to.key`
const resolveAttribute = (obj, key) => key
  .split('.')
  .reduce((prev, curr) => prev?.[curr], obj);

// Recursively builds JSX output adding `<mark>` tags around matches
const highlight = (value, indices = [], i = 1) => {
  const pair = indices[indices.length - i];
  return !pair ? value : (
    <>
      {highlight(value.substring(0, pair[0]), indices, i+1)}
      <mark>{value.substring(pair[0], pair[1]+1)}</mark>
      {value.substring(pair[1]+1)}
    </>
  );
};

// FuseHighlight component
const FuseHighlight = ({ hit, attribute }) => {
  const matches = typeof hit.item === 'string'
    ? hit.matches?.[0]
    : hit.matches?.find(m => m.key === attribute);
  const fallback = typeof hit.item === 'string'
    ? hit.item
    : resolveAttribute(hit.item, attribute);
  return highlight(matches?.value || fallback, matches?.indices);
};

export default FuseHighlight;

By extracting the helper functions from the components, I ensure they are not recomputed each time the component renders, so basically at (almost) each key press.

A few things to take into consideration:

Fuse handles both string array and object array searches. So we have to ensure that our highlighting still works with string array searches. In that case, there's no need to pass the attribute argument, as the string value will be stored directly into hit.item and its matches will be in hit.matches[0] without a key attribute.
When there is no matches indices (empty query), we still want to return the whole string value. In that case we need to find that value in the original item data, and we do so using our resolveAttribute helper.
We could have built a string with HTML markup and pass it inside a <span> with the dangerouslySetinnerHTML attribute. I did at first, but it adds an unnecessary DOM element. We're in a JSX file, so let's take the most out of it.
The highlight helper relies heavily on the fact that indices are sorted in ascending order and have no overlaps. That way, we just extract the last match (indices.pop()), directly wraps the markup around it, and recursively apply the same instructions to the remaining beginning of the string value.

We can now complete our example, and it's as simple as that:

// 😎 Bring it on!
import React from 'react';
import FuseHighlight from 'components/FuseHighlight';
import { useFuse } from 'features/useFuse';
const MySearch = (props) => {
  const { hits, query, onSearch } = useFuse(props.list, {
    keys: ['name'],
    includeMatches: true,
    matchAllOnEmptyQuery: true,
  });
  return (
    <input
      name="search"
      type="search"
      placeholder="Search..."
      autoComplete="off"
      onKeyUp={onSearch}
      onChange={onSearch} // handles "clear search" click
    />
    <p>Results for "{query}":</p>
    <ol>
      {hits.map(hit => (
        <li key={hit.refIndex}>
          <FuseHighlight
            hit={hit}
            attribute="name"
          />
        </li>
      ))}
    </ol>
  );
};

We have created two elegant hook and component that completely enhance our developer experience while preserving the performances. They have a very limited footprint, and yet we have everything we need to build a nice advanced search into our applications.

Unify ACL across front-end and back-end with Next.js and NextAuth.js

Nicolas Torres — Thu, 14 Jan 2021 09:13:42 +0000

Access Control List (ACL) is a security layer to restrict access to resources based on the user's or client's assigned role. While it's being widely used to protect APIs, it can be convenient to also hide some features on the interface from unauthorized users. Instead of duplicating the permissions definition and logic, we'll see how we can seamlessly combine and share them across the application front-end and back-end with Next.js.

Note that implementing ACL on the front-end only is not enough of a security. An advised user could inject the permissions and reveal any protected feature you're trying to hide from them. That's why it's important to protect both ends of the application.

Before we start

In this article, I'll use the AccessControl package to implement ACL because I like its structure, performances, and simple usage, but the example can be replicated with other ACL packages as well.

I also heavily rely on next-connect to chain middlewares. Again, it's not mandatory, but it makes the API handlers less verbose and therefore shorten the examples.

The folder structure I've set is completely arbitrary apart from pages which is the entry point in Next.js. I share it there in order to clarify the examples:

+-- public/                 // Static assets
+-- src/                    // Whole application source code
|   +-- app/                // Front-end utilities
|   |   +-- components/     // Base resuable components
|   |   +-- features/       // Custom hooks, helpers, contexts
|   |   +-- layouts/        // App specific components
|   +-- pages/              // Entry point for Pages
|   |   +-- api/            // Entry point for API routes
|   +-- server/             // Back-end utilities
|   |   +-- helpers/        // Helper functions
|   |   +-- middlewares/    // API middleware functions
|   |   +-- services/       // Third-party services instances

1. Defining the permissions

AccessControl stores permissions into memory. We'll define permissions in a module so they're centralized, and loaded any time we require the module for the first time after the process starts.

// src/server/services/accesscontrol.js
// ----------------------------------------
import AccessControl from 'accesscontrol';

// unique back-end instance of AccessControl
export const ac = new AccessControl();

// owners can manage all users
ac.grant('owner').resource('users').readAny().createAny().updateAny().deleteAny();

// guests can manage only their own profile
ac.grant('guest').resource('users').readOwn().updateOwn();

// we lock ACL to avoid updates out of this file
ac.lock();

// src/server/services/index.js
// ----------------------------------------
export { ac } from './accesscontrol';

2. Protecting API routes

The basic way to check a permission is running:

const permission = ac.can('guest').readOwn('users');
console.log(permission.granted); // -> true

// or
const permission = ac.permission({
  role: 'guest',
  resource: 'users',
  action: 'read:own',
)};
console.log(permission.granted); // -> true

But having to pass the role each time we ask for a permission is a bit verbose and unnecessary. It's up to us to implement a middleware the way we want it, and I usually store the user's role in session. Let's create a checkAccess middleware that will read the role from the session so there's no need to specify it at each call.

// src/server/middlewares/checkAccess.js
// ----------------------------------------
import { ac } from 'server/services'; // load ACL

export const checkAccess = (resource, action, possession) => (req, res, next) => {
  let permission;
  try {
    permission = ac.permission({
      role: req.session?.user?.role, // we'll see later about that
      resource,
      action,
      possession,
    });
  } catch { // `ac.permission` throws if role is not a string
    permission = { granted: false };
  }

  // return 403 if access is denied
  if (!permission.granted) {
    return res.status(403).json({
      ok: false,
      message: 'You are not authorized to access this resource',
    });
  }

  return next();
};

// src/server/middlewares/userAuth.js
// ----------------------------------------
import { getSession } from 'next-auth/client';

// check if user is authenticated
export const userAuth = async (req, res, next) => {
  // store session into request to pass it to following middlewares
  req.session = await getSession({ req });
  if (!req.session) {
    return res.status(403).send({
      ok: false,
      message: `User authentication required.`,
    });
  }
  return next();
};

// src/server/middlewares/index.js
// ----------------------------------------
export { checkAccess } from './checkAccess';
export { userAuth } from './userAuth';

Then we can simply chain it before our final API route handler to protect its access:

// src/page/api/users/[id].js
// ----------------------------------------
import nc from 'next-connect';
import { checkAccess, userAuth } from 'server/middlewares';

const handler = nc();
  .use(userAuth) // injects session into req.session
  .use(checkAccess('users', 'read:own'))
  .use(checkAccess('profile', 'read:own')) // you can chain multiple checks
  .get((req, res) => {
    // get user profile from DB
    return res.send({
      ok: true,
      data: user,
    });
  });

export default handler;

What about specific permissions per method?

// src/page/api/users/[id].js
// ----------------------------------------
import nc from 'next-connect';
import { checkAccess, userAuth } from 'server/middlewares';

const handler = nc();
  .use(userAuth) // injects session into req.session
  .get(checkAccess('users', 'read:own'), (req, res) => {
    // get user profile from DB
    return res.send({
      ok: true,
      data: user,
    });
  })
  .put(checkAccess('users', 'update:own'), (req, res) => {
    // update user profile in DB
    return res.send({
      ok: true,
      data: user,
    });
  })
  .delete(checkAccess('users', 'delete:any'), (req, res) => {
    // delete user profile from DB
    return res.send({
      ok: true,
      data: {},
    });
  });

export default handler;

3. Adding user role to the session

NextAuth makes it a breeze to handle user authentication with both email and social accounts. Though we'll need to extend it a little bit to add a role column into the users database table it creates, and then inject the role into the session. Following their documentation, we end up doing:

// src/page/api/[...nextauth].js
// ----------------------------------------
import NextAuth from 'next-auth';
import Adapters from 'next-auth/adapters';
import Providers from 'next-auth/providers';

const options = {
  site: process.env.NEXTAUTH_URL,
  database: process.env.DATABASE_URL,
  providers: [/* ... */],
  // alter user schema to add a `role` column
  adapter: Adapters.TypeORM.Adapter(
    process.env.DATABASE_URL,
    {
      models: {
        User: {
          model: Adapters.TypeORM.Models.User.model,
          schema: {
            ...Adapters.TypeORM.Models.User.schema,
            columns: {
              ...Adapters.TypeORM.Models.User.schema.columns,
              role: {
                type: 'varchar',
                nullable: true,
              },
            }
          }
        },
      },
    }
  ),
  // assign default role on user creation
  events: {
    createUser: async (user) => {
      // use your database package to safely perform the following query:
      await db.run(
        'UPDATE users SET role="guest" WHERE id=${id}',
        { id: user.id }
      );
    },
  },
  // enrich session data
  callbacks: {
    session: async (session, user) => {
      session.user.role = user.role; // inject role into session
      return Promise.resolve(session);
    },
  },
};

export default (req, res) => NextAuth(req, res, options);

4. Sharing permissions with the front-end

To check permissions on the front-end, we could technically just require src/server/services/accesscontrol.js and perform the same checks using ac.can('guest').readOwn('users').granted for example. But as we're in the browser and already using React, better spare as much memory as we can and not load the full list of permissions, that can be very long in large applications.

That's why the strategy here is to pass only the permissions tied to the user's role, and for this, we'll again rely on the session object. Let's update our session callback in NextAuth options:

// src/page/api/[...nextauth].js
// ----------------------------------------
// ...
import { ac } from 'server/services';
// ...
const options = {
  // ...
  // enrich session data
  callbacks: {
    session: async (session, user) => {
      session.user.role = user.role;
      // get all permissions
      const grants = ac.getGrants();
      // expose only the current role permissions
      session.user.permissions = user.role in grants ?
        { [user.role]: grants[user.role] } :
        {};
      return Promise.resolve(session);
    },
  },
};
// ...

We'll then create an independent and volatile instance of AccessControl in the front-end, and store the permissions into a React context to check access from any component.

4. Storing permissions into a context

Again here, to avoid passing the role as argument each time to AccessControl, we'll expose a nice wrapper from the React context we'll create for accessing user's data:

// src/app/features/UserContext.jsx
// ----------------------------------------
import AccessControl from 'accesscontrol';
import { useSession } from 'next-auth/client';
import React, { createContext, useCallback, useContext, useEffect } from 'react';

// front-end isolated instance of AccessControl
const ac = new AccessControl();

// create and export User context
export const UserContext = createContext({
  access: () => ({ granted: false }),
  isLoading: false,
  user: {},
});

// export hook context wrapper
export const useUserContext = () => useContext(UserContext);

// export pre-configured provider
export const UserProvider = (props) => {
  const [session, isLoading] = useSession();

  // update permissions from session data
  const permissions = session?.user?.permissions || {};
  useEffect(
    () => {
      ac.setGrants(permissions);
      return () => {
        ac.reset(); // reset when permissions change
      };
    },
    [permissions]
  );

  // expose access checking wrapper method
  // automatically fill user role on permission check
  const role = session?.user?.role || 'guest';
  const access = useCallback(
    (resource, action, possession) => {
      try {
        return ac.permission({
          role,
          resource,
          action,
          possession,
        });
      } catch { // if role is not a string
        return { granted: false };
      }
    },
    [role]
  );

  // Return context values
  // ----------------------------------------
  const context = {
    access,
    isLoading,
    user: session?.user,
  };

  return (
    <UserContext.Provider value={context}>
      {props.children}
    </UserContext.Provider>
  );
};

// src/app/features/index.js
// ----------------------------------------
export { UserContext, useUserContext } from './UserContext';

Then we only need to wrap our application with the User context inside src/pages/_app.js:

// src/pages/_app.js
// ----------------------------------------
import React from 'react';

import { UserProvider } from 'app/features';

const App = ({ Component, pageProps }) => (
  <UserProvider>
    <Component {...pageProps} />
  </UserProvider>
);

export default App;

5. Hiding features from unauthorized users

The setup is now complete, we have protected our API routes and the users, even if they hack their way to access the features from the interface, they won't be able to perform any action on the API as long as your permissions are defined accordingly.

So, hiding content on the interface becomes a matter of seconds using our User context, for example to filter the list of accessible links in the main navigation:

// src/app/layouts/MainNav.jsx
// ----------------------------------------
import Link from 'next/link';
import { useUserContext } from 'app/features';

const MainNav = (props) => {
  const router = useRouter();
  const { access } = useUserContext();

  return (
    <ul>
      <li>
        <Link href="/">
          <a>Dashboard</a>
        </Link>
      </li>
      {access('users', 'read:any').granted && (
        <li>
          <Link href="/users">
            <a>Manage users</a>
          </Link>
        </li>
      )}
      {access('users', 'read:own').granted && (
        <li>
          <Link href="/profile">
            <a>Profile</a>
          </Link>
        </li>
      )}
    </ul>
  );
};

The code examples may seem quite long, but we've basically just wrapped some AccessControl features to provide a better developer experience within our stack, ending up with a unified and performant solution to protect both front-end and back-end features using a single ACL declaration. Once the setup complete, the usage is a matter of adding a single line of code here and there.

If you're not using any of the packages here, it's still quite easy to replicate the strategy with vanilla Next.js and whichever session manager you use. I personally also assign a role to any API client (the front-end is a client too) based on the Bearer Token they're using (into req.role directly, overriden when there's a user session with role: req.session?.user?.role || req.role in checkAccess middleware). That way I can open some API routes for external user-less usage and limit their scope with the same ACL layer.

Build a full API with Next.js

Nicolas Torres — Mon, 21 Dec 2020 19:35:47 +0000

After years and years fighting with and against JavaScript build stacks, I eventually gave a try to Next.js and fell in love with it for two simple reasons: it's barely opinionated, and it packages a simple and unique build configuration shared across back-end and front-end. But as it's not Express underneath the API routes, we have to find some workarounds to be able to build a real all-in-one application.

To call it a decent API, we need quite more than just routes handling. Standalone entry points are required for executing scripts and workers; chaining middlewares really helps keeping route security layers declaration succinct; and as most middlewares and router-dependent packages have been written for Express, we also need a way to integrate them seamlessly.

One solution would be using a custom Express server, but we'd go against the framework and lose its main advantage: Automatic Static Optimization. So let's try to use the built-in server, and address the issues one by one to make it all run smooth.

Issue 1: chaining middlewares

This one is a no-brainer. Just use next-connect! It emulates the next() behavior of Express and gives us back our well-appreciated .use(), .get(), .post(), .all() etc. methods that takes away the need for the verbose in-route method checking (if (req.method === 'POST') { ... }) that Next.js suggests on their documentation.

import nc from 'next-connect';

const handler = nc()
  .use(someMiddleware())
  .get((req, res) => {
    res.send('Hello world');
  })
  .post((req, res) => {
    res.json({ hello: 'world' });
  });

export default handler;

Also, a very convenient feature is passing other next-connect instances to the .use() method, and therefore predefine reusable handler middlewares:

// /path/to/handlers.js
import nc from 'next-connect';
import { acl, apiLimiter, bearerAuth } from '/path/to/middlewares';

export const baseHandler = () => nc({
  // 404 error handler
  onNoMatch: (req, res) => res.status(404).send({
    message: `API route not found: ${req.url}`,
  }),

  // 500 error handler
  onError: (err, req, res) => res.status(500).send({
    message: `Unexpected error.`,
    error: err.toString(),
  }),
});

export const secureHandler = baseHandler()
  .use(apiLimiter)
  .use(bearerAuth)
  .use(acl);


// /pages/api/index.js
import nc from 'next-connect';
import { secureHandler } from '/path/to/handlers';
const handler = nc()
  .use(secureHandler) // benefits from all above middlewares
  .get((req, res) => {
    res.send('Hello world');
  });
export default handler;

Issue 2: testing routes

Within the test environment, Next.js server is not running, forcing us to find a way to emulate both the request and its resolution. Supertest pairs really well with Express, but needs to run the server in order to pass the request to the handler through all its layers. That being said, it doesn't need to be Express.
So without adding any new dependency, we create a bare HTTP server with the native node http lib, and manually apply the built-in resolver of Next.js, nicely packaged as a utility function, just like this:

import { createServer } from 'http';
import { apiResolver } from 'next/dist/next-server/server/api-utils';
import request from 'supertest';

export const testClient = (handler) => request(httpCreateServer(
  async (req, res) => {
    return apiResolver(req, res, undefined, handler);
  },
));

In our test files, the only thing we need then is passing the handler to our client, with Supertest running as usual:

import { testClient } from '/path/to/testClient';
import handler from '/pages/api/index.js';

describe('/api', () => {
  it('should deny access when not authenticated', async (done) => {
    const request = testClient(handler);
    const res = await request.get('/api');
    expect(res.status).toBe(401);
    expect(res.body.ok).toBeFalsy();
    done();
  });
});

That way we don't have anything to setup repeatedly for each route test. Pretty elegant.

Issue 3: custom entry points

Entry points are scripts that are meant to be run manually - usually background processes like a queue worker, or migration scripts. If set as standalone node processes, they won't inherit from the 'import' syntax built-in inside Next.js, neither the path aliases you may have setup. So basically, you'd have to manually rebuild the build stack of Next.js, polluting your package.json with babel dependencies, and keep it up to date with Next.js releases. We don't want that.

To make it clean, we have to make these pipe through Next.js build. Adding custom entry points is not documented, though it seems to work with that solution, configuring next.config.js:

const path = require('path');

module.exports = {
  webpack(config, { isServer }) {
    if (isServer) {
      return {
        ...config,
        entry() {
          return config.entry().then((entry) => ({
            ...entry,
            // your custom entry points
            worker: path.resolve(process.cwd(), 'src/worker.js'),
            run: path.resolve(process.cwd(), 'src/run.js'),
          }));
        }
      };
    }
    return config;
  },
};

Sadly the only thing it does is compiling these new JavaScript files through the internal webpack process and outputs them inside the build directory, as is. Since they're not tied to the server, all the features of Next.js are missing, including the only important one for this case: environment variables.

Next.js relies on dotenv, so it's already set as a dependency that we could reuse. Yet calling dotenv at the top of these entry points, for some reasons, won't propagate the environment variables to the imported modules:

// /.env
FOO='bar';


// /src/worker.js
import dotenv from 'dotenv';
dotenv.config();

import '/path/to/module';

console.log(process.env.FOO); // outputs 'bar';


// /src/path/to/module.js
console.log(process.env.FOO); // outputs 'undefined';

That is very annoying. Thankfully, it can be quickly solved by dotenv-cli, which actually resolves .env files the same way than Next.js. We only need to prefix our script commands in package.json:

"worker": "dotenv -c -- node .next/server/worker.js",

Note that it calls the script from the build folder. You need either to have next dev running, or previously have run next build. It's a small price to pay in regard of the benefits of keeping them within the Next.js build stack.

Issue 4: Express-based packages

Next-connect already makes some Express packages compatible out of the box, like express-validator that I'm used to when it comes to checking request parameters. That's because they simply are middleware functions.

Some of these functions rely on Express-specific properties, like express-acl. Usually they throw an exception when hitting that missing property, and digging a little bit the error and the package source will help you find it and fix it with a handler wrapper:

import acl from 'express-acl';

acl.config({
  baseUrl: '/api',
  filename: 'acl.json',
  path: '/path/to/config/folder',
  denyCallback: (res) => res.status(403).json({
    ok: false,
    message: 'You are not authorized to access this resource',
  }),
});

export const aclMiddleware = (req, res, next) => {
  req.originalUrl = req.url; // Express-specific property required by express-acl
  return acl.authorize(req, res, next);
};

So the biggest challenge happens when the package deeply depends on Express because it creates router or app definitions. That's the case of monitoring interfaces like bull-board. When we can't find a standalone alternative, then our only chance is to find a way to emulate the whole Express application. Here's the hack:

import Queue from 'bull';
import { setQueues, BullAdapter, router } from 'bull-board';
import nc from 'next-connect';

setQueues([
  new BullAdapter(new Queue('main')),
]);

// tell Express app to prefix all paths
router.use('/api/monitoring', router._router);

// Forward Next.js request to Express app
const handler = nc();
handler.use((req, res, next) => {
  // manually execute Express route
  return router._router.handle(req, res, next);
});

export default handler;

A few things to note here:

This file should be located inside /pages/api because Next.js only recognize server-side routes under that folder.
For Express to handle all sub-routes declared by the package, we have to create a catch-all on Next.js route. That can be done naming our route file /pages/api/monitoring/[[...path]].js as specified in their docs (replace "monitoring" with whichever name you'd prefer).
In this specific case, bull-board exposes an entire Express instance under the confusing name router. That's why we're calling router._router.handle() to manually execute the route handler. If by reading the source you find out it's a express.Router instance, call instead router.handle() directly.
We also need to tell Express that the base URL of its entire app is the route we're calling it from. Let's just define it with app.use('/base/url', router) as we would normally do. Just keep in mind the confusion between express and express.Router instances.
Finally, Express handles the response part as we're passing it the full Response object. No need for us to send headers on its behalf.

The reasons why I don't use this trick to forward the whole API to an emulated Express app is that I don't know how it'll affect performances, and most importantly, I'd rather respect Next.js natural patterns not to disorient other developers.

Not so bad, isn't it? We end up having a full-featured server with footprint-limited patches over the blind spots. I still wish Next.js could provide all these features in its core, but I'm happy we didn't denature it much either with these workarounds. Given the current state of JavaScript, Next.js may very well be the ultimate full-stack framework.

PS: I didn't go over setting up sessions and user authentication because with these issues now solved, you can virtually make everything work as usual. Though, I'd recommend looking into next-session or NextAuth.js.

DEV Community: Nicolas Torres

How to use DEV.to as a CMS to make a blog with NextJS

1. Preparing NextJS

Dependencies

Environment variables

Pages structure with /pages router

Pages structure with /app router

2. Preparing Methods

3. Listing articles

/pages router

/app router

Link articles

4. Article page

/pages router, SSR mode

/pages router, SSG mode

/app router

Adding anchors and code snippets syntax highlighting

5. Updating canonical for SEO

6. Bonus: handling pagination

Folder structure

Adding pagination data to articles list

/pages router, SSR mode

/pages router, SSG mode

/app router

Introducing Live Patching

Motivation

How does it work

Using the admin panel

Using the API

Adding a missing location

Fixing an existing location

Listing patch records

Retrieving a patch record

Updating a patch record

Delete a patch record

Making React-Leaflet work with NextJS

Passing ref

Organizing your components

Using custom Marker icons

Handling map events

Run a worker alongside Next.js server using a single command

Fix "too many connections" errors with database clients stacking in dev mode with Next.js

Bypassing Shopify Admin REST API limitations with a custom JS client

Usage

Client source code

Using Fuse.js with React to build an advanced search (with highlighting)

Creating a useFuse custom hook

Highlighting results

Unify ACL across front-end and back-end with Next.js and NextAuth.js

Before we start

1. Defining the permissions

2. Protecting API routes

3. Adding user role to the session

4. Sharing permissions with the front-end

4. Storing permissions into a context

5. Hiding features from unauthorized users

Build a full API with Next.js

Issue 1: chaining middlewares

Issue 2: testing routes

Issue 3: custom entry points

Issue 4: Express-based packages

Pages structure with `/pages` router

Pages structure with `/app` router

`/pages` router

`/app` router

`/pages` router, SSR mode

`/pages` router, SSG mode

`/app` router

`/pages` router, SSR mode

`/pages` router, SSG mode

`/app` router

Passing `ref`

Creating a `useFuse` custom hook