DEV Community: Alan P John

Forking OpenFaaS Faasd to support Firecracker Containerd

Alan P John — Mon, 10 Jun 2024 15:35:07 +0000

Why you would need to do that?

As part of my research, I needed to evaluate the performance of Firecracker in serverless environments compared to traditional Linux containers (LXC). OpenFaaS, with its modular design, offered an excellent framework for this comparison. OpenFaas offered two running modes, which were OpenFaas using Kubernetes and faasd. Firecracker-containerd isn’t directly supported by Kubernetes due to the lack of a stable CRI plugin unless you consider the now unsupported Firekube. Extending faasd to support Firecracker was simpler and served as sufficient proof of concept from my research. Otherwise, from a general point of view, the primary advantage of Firecracker over LXC in serverless computing is isolation, which isn’t crucial if you’re running faasd since serverless loads on faasd are typically trusted. So, there is no big need to do this other than plain curiosity.

With that in mind, let’s dive into the details. This guide isn’t a polished product; there are still rough edges, which I’ll cover in the conclusion section later.

Prerequisites

To set up Firecracker, I followed the guide at Firecracker-containerd's Getting Started. The key steps involve building the necessary components by running the make command.

Before running make, I made a few adjustments to the Makefile to use golang:1.21-bullseye as the builder image. Then, I ran the following commands:

make all to get firecracker-ctr, firecracker-containerd, and containerd-shim-v2-firecracker.
make firecracker to get the Firecracker binary
make image to get the rootfs

The goal is to have binaries for Firecracker, firecracker-ctr, firecracker-containerd, and containerd-shim-v2-firecracker, along with the Firecracker CNI plugin tc-redirect-tap. The guide well explains each setup.

I use the tc-redirect-tap CNI plugin to connect containers running on Firecracker to the CNI network. A five-year-old (quite outdated) guide on Firecracker-containerd CNI networking options is found here. For simplicity's sake, I connect to the instances without using a network bridge and let Firecracker-containerd set up the networks. I attempted to adjust the configuration to use a network bridge, but using the bridge CNI plugin with the tc-redirect-tap plugin shows some weird behaviour, ultimately making the instance unreachable. So we set up container networking through the firecracker containerd configuration as mentioned in the Networking Support section of the getting started guide. so ensure that your firecracker-containerd configuration files /etc/containerd/firecracker-runtime.json and config.toml are configured as per as the getting started guide.
I recommend testing out your firecracker-containerd setup and playing around with it to get comfortable with it.

Extending faasd to firecracker-containerd

I had to set faasd to use the firecracker-containerd socket instead of the default containerd socket. While we can pass the socket and runtime using environment variables, as previously mentioned, the objective is only running on firecracker containerd, so we can hard set the default runtime and socket. Changing the runtime and socket at runtime using environment variables would mean adding code to handle both cases and validating the configurations of all components, which is additionaly code which we are avoiding for now. You can override the default runtime in the ReadFromEnv function at the pkg/provider/config/read.go. Changes need to be made to the read_test.go test file accordingly.

func ReadFromEnv(hasEnv types.HasEnv) (*types.FaaSConfig, *ProviderConfig, error) {
    //
    // ... rest of the code is untouched
    //
    providerConfig := &ProviderConfig{
        Sock: types.ParseString(hasEnv.Getenv("sock"), "/run/firecracker-containerd/containerd.sock"),
    }
    return config, providerConfig, nil
}

Setting default runtime for containerd at cmd/provider.go. runtime to aws.firecracker.

func runProviderE(cmd *cobra.Command, _ []string) error {
    //
    // ... rest of the code is untouched
    //
    client, err := containerd.New(
        providerConfig.Sock,
        containerd.WithDefaultRuntime("aws.firecracker"),
    )
    //
    // ... 
    //
}

Changes to instance management

Swapping containerd for firecracker-containerd isn't straightforward and requires changes to the container creation code for faasd in pkg/provider/handlers/deploy.go. This part was a little tricky, and I must admit, I feel I might be missing a trick, but I did get it to work as intended with minimal changes, so I am going ahead and committing to it.
We would need to add firecracker-containerd as a dependency to the project

$ go get github.com/firecracker-microvm/firecracker-containerd/

and import the firecrackeroci submodule

import (
    // ... other imports
    "github.com/firecracker-microvm/firecracker-containerd/runtime/firecrackeroci"
)

Now, we adjust the container options for firecracker-containerd by adding the firecrackeroci.WithVMID and firecrackeroci.WithVMNetwork options to be able to identify the VM and make firecracker containerd setup the networking as per the previous discussion on networking. We also commented out the mounts option as adding mounted volumes on firecracker-containerd was causing permission issues, and firecracker-containerd could not find the mounts (An issue that is persistent when running with Microk8s Kata as well).

func deploy(ctx context.Context, req types.FunctionDeployment, client *containerd.Client, cni gocni.CNI, secretMountPath string, alwaysPull bool) error {
    //
    // ... rest of the code is untouched
    //
    copts := []containerd.NewContainerOpts{
        containerd.WithImage(image),
        containerd.WithSnapshotter(snapshotter),
        containerd.WithNewSnapshot(name+"-snapshot", image),
        containerd.WithNewSpec(
            oci.WithImageConfig(image),
            oci.WithCapabilities([]string{"CAP_NET_RAW"}),
            // oci.WithMounts(mounts),
            oci.WithEnv(envs),
            firecrackeroci.WithVMID(name),
            firecrackeroci.WithVMNetwork,
            withMemory(memory)),
        containerd.WithContainerLabels(labels),
    }

    container, err := client.NewContainer(
        ctx,
        name,
        copts...,
    )
    //
    // ...
    //
}

Attaching the container IO to the faasd binary (which is faasd default behaviour) does not work with firecracker containerd, so we attach it to a temporary log file created for the container for now. Additionally, as firecracker containerd creates the CNI network for the function, we dont need to create it again here.

func createTask(ctx context.Context, container containerd.Container, _ gocni.CNI) error {
    name := container.ID()
    task, taskErr := container.NewTask(ctx, cio.LogFile(fmt.Sprintf("/tmp/%s.log", name)))
    if taskErr != nil {
        return fmt.Errorf("unable to start task: %s, error: %w", name, taskErr)
    }
    log.Printf("Container ID: %s\tTask ID %s:\tTask PID: %d\t\n", name, task.ID(), task.Pid())
    _, waitErr := task.Wait(ctx)
    if waitErr != nil {
        return errors.Wrapf(waitErr, "Unable to wait for task to start: %s", name)
    }
    if startErr := task.Start(ctx); startErr != nil {
        return errors.Wrapf(startErr, "Unable to start task: %s", name)
    }
    ip, err := cninetwork.GetIPAddress(name)
    if err != nil {
        return err
    }
    log.Printf("%s has IP: %s.\n", name, ip)
    return nil
}

You will notice that the cninetwork.GetIPAddress call has changed; we will get to this in the next section.
As you would have set which following the firecracker-containerd getting started guide, We would use the devmapper snapshotter. We will set the default snapshotter to devmapper, which would require changing the snapshotter:= "" lines to snapshotter:= "devmapper" in the deploy and prepull functions.

func Remove(ctx context.Context, client *containerd.Client, name string) error {
    //
    // ...
    //
    } else {
        snapshotter := "devmapper"
        if val, ok := os.LookupEnv("snapshotter"); ok {
            snapshotter = val
        }
        service := client.SnapshotService(snapshotter)
        key := name + "-snapshot"
        if _, err := client.SnapshotService("").Stat(ctx, key); err == nil {
            service.Remove(ctx, key)
        }
    }
    return nil
}

Changes to networking

As the networks set up by firecracker containerd are ptp and setup by firecracker containerd and not by us, there are some major changes in the cni_network submodule. Firstly, we need to set up the CNI constants in line with the configuration passed to firecracker containerd.

const (
    // CNIBinDir describes the directory where the CNI binaries are stored
    CNIBinDir = "/opt/cni/bin"
    // CNIConfDir describes the directory where the CNI plugin's configuration is stored
    CNIConfDir = "/etc/cni/conf.d"
    // NetNSPathFmt gives the path to the a process network namespace, given the pid
    NetNSPathFmt = "/proc/%d/ns/net"
    // CNIDataDir is the directory CNI stores allocated IP for containers
    CNIDataDir = "/var/run/cni"
    // defaultCNIConfFilename is the vanity filename of default CNI configuration file
    defaultCNIConfFilename = "fcnet.conflist"
    // This value appears in iptables comments created by CNI.
    defaultNetworkName = "fcnet"
    // defaultSubnet is the default subnet used in the defaultCNIConf -- this value is set to not collide with common container networking subnets:
    defaultSubnet = "10.64.0.0/16"
    // defaultIfPrefix is the interface name to be created in the container
    defaultIfPrefix = "veth"
)

// defaultCNIConf is a CNI configuration that enables network access to containers
var defaultCNIConf = fmt.Sprintf(`
{
    "cniVersion": "0.4.0",
    "name": "%s",
    "plugins": [
      {
        "type": "ptp",
        "ipMasq": true,
        "ipam": {
            "type": "host-local",
            "subnet": "%s",
            "dataDir": "%s",
            "routes": [
                { "dst": "0.0.0.0/0" }
            ]
        }
      },
      {
        "type": "firewall"
      },
      {
        "type": "tc-redirect-tap"
      }
    ]
}
`, defaultNetworkName, defaultSubnet, CNIDataDir)

faasd retrieves the IP addresses for the containers by using the service name and the container task PID. However, I observed the task PID returned by the firecracker-containerd is not in line with the task PID of the firecracker process, which means the current code to retrieve the IP address would fail. So, I removed the PID parameter and used the service name to retrieve the IP address.

func isCNIResultForContainer(fileName, container string) (bool, error) {
    found := false

    f, err := os.Open(fileName)
    if err != nil {
        return false, fmt.Errorf("failed to open CNI IP file for %s: %v", fileName, err)
    }
    defer f.Close()

    reader := bufio.NewReader(f)
    processLine, _ := reader.ReadString('\n')
    if strings.Contains(processLine, container) {
        ethNameLine, _ := reader.ReadString('\n')
        if strings.Contains(ethNameLine, defaultIfPrefix) {
            found = true
        }
    }

    return found, nil
}

Thus we no longer need the PID to retrieve the IP address

func GetIPAddress(container string) (string, error) {
    CNIDir := path.Join(CNIDataDir, defaultNetworkName)

    files, err := os.ReadDir(CNIDir)
    if err != nil {
        return "", fmt.Errorf("failed to read CNI dir for container %s: %v", container, err)
    }
    for _, file := range files {
        // each fileName is an IP address
        fileName := file.Name()

        resultsFile := filepath.Join(CNIDir, fileName)
        found, err := isCNIResultForPID(resultsFile, container)
        if err != nil {
            return "", err
        }
        if found {
            return fileName, nil
        }
    }
    return "", fmt.Errorf("unable to get IP address for container: %s", container)
}

Changes must be made to the cni_network_test.go test file accordingly.
The GetIPAdress function is also called the pkg/supervisor.go, which runs the Faas gateway and other processes required by OpenFaas. Therefore, we would need to modify the function call in that.

Changes to conf files

To avoid this fork from clashing with an existing OpenFaas faasd installation, I renamed the binary and systemd services as faasd-fc; in the relevant file paths, in the Makefile, the systemd service files in hack/faasd-fc.service and hack/faasd-fc-provider.service and the cmd/install.go file.
You can build the faasd-fc binary using the command.

make dist-local

Running It

You can start faasd-fc by running

sudo bin/faasd-fc install

Follow the instruction in the command output to setup faas-cli.

$ faas-cli ls
Function                        Invocations     Replicas

I created a simple hello-world application in go to test the deployment. Lets deploy the function using faas-cli.

$ faas-cli deploy --image=alanjohn/hello-go:latest --name=hello-go --update=false   
Function hello-go already exists, attempting rolling-update.

Deployed. 200 OK.
URL: http://127.0.0.1:8080/function/hello-go

You can use firecracker-ctr in the openfaas-fn namespace to check if the container and task for this container are running with the correct runtime and configuration. We can use curl to interact with the application.

$ curl http://127.0.0.1:8080/function/hello-go
hello, world!
$ faas-cli ls
Function                        Invocations     Replicas
hello-go                        1               1

It is working as expected
Now lets delete this function

$ faas-cli rm hello-go
Deleting: hello-go.
Removing old function.
$ faas-cli ls         
Function                        Invocations     Replicas

We can use firecracker-ctr to confirm that all resources in openfaas-fn namespace have been cleaned up.

Conclusion

You can find the full code here. Ideally, to get the most out of firecracker-containerd, you would want to have the firecracker instances running beforehand to remove the boot time of the firecracker VM from the time taken to start an instance as done by AWS lambda who have running firecracker microVM slots. But that would require some major refactoring to faasd and would be an overkill to faasd's intended use case. The lack of a network bridge and volume support is a more imminent issue that would completely bridge the gap between faasd and this firecracker-container extension that uses firecracker-containerd. Open to some inputs on how I could go around those.

Understanding SEO and Web Vitals for your NextJS site and how to improve them?

Alan P John — Sun, 01 May 2022 11:29:29 +0000

Prerequisites

This article is part of a series where I break down the process of building my developer portfolio. The previous 2 parts focussed on how I setup my portfolio and built it using Notion API as a headless CMS. Now that the portfolio and blog was ready, it was time to make it accessible to the public. To make sure that people are able to find and search your site organically, we need to have our site pop-up high on search engine results. This is where Search Engine Optimization comes into the picture.

Search Engine Optimization

Search Engine Optimization can be defined as making our content more search engine friendly so that relevant users can find it more easily. This would include:

Adding meta Tags

Meta tags exist in the head element of the page providing important information about the page that can used by search engines and social media sites. There are a variety of meta tags that represent different types of information for different search engines. A popular subset of standardized meta tags are described in the open graph protocol. Managing these tags for every page and generating them dynamically for blog pages has it’s own overhead. Thus for that reason, I have used the Next-SEO plugin. The Next-SEO plugin provides DefaultSEO component that contains tags that will be defined for all routes by default which subsequently can be overwritten at a page level by then using the NextSEO component. Below is the default SEO config used by me.

export const Layout: React.FC<Props> = ({ children }: Props) => {
    const router = useRouter();
    const url = `${domain}${router.asPath}`;
    return (
        <div className="flex w-full flex-col min-h-screen">
            <DefaultSeo
                titleTemplate="%s - Alan John"
                openGraph={{
                    type: "website",
                    locale: "en_IE",
                    url: url,
                    description:
                        "The personal website for Alan John, developer.",
                    site_name: "Alan John",
                    images: [
                        {
                            url: `${domain}/images/social_media_preview.png`,
                            width: 1200,
                            height: 628,
                            alt: "My Portfolio Preview",
                            type: "image/png",
                        },
                    ],
                }}
                canonical={url}
            />
            <Header />
            {children}
            <Footer />
        </div>
    );
};

And this is how I have overwritten only the tags that require to be changes at the page level for the home page.

const Home: NextPage = () => {
    return (
        <Layout>
            <NextSeo
                title="Home"
                description="2022 Portfolio"
                additionalMetaTags={[
                    {
                        property: "keywords",
                        content:
                            "Alan, John, Software Developer, Sofware Engineer, Developer, Portfolio",
                    },
                ]}
            />
            <Section className="mt-32 flex-grow">
                        {/* Contents of Page */}
            </Section>
        </Layout>
    );
};

So when we inspect the Head element of the page, we get all the meta tags we would require for our page.

The contents of the meta tags has to studied properly to get best results. There are plenty of resources talking about great SEO strategies. The contents in the tags I have added are just for demonstration.

<meta name="viewport" content="width=device-width">
<meta charset="utf-8">
<meta property="og:type" content="website">
<meta property="og:image:alt" content="My Portfolio Preview">
<meta property="og:image:type" content="image/png">
<meta property="og:image:width" content="1200">
<meta property="og:image:height" content="628">
<meta property="og:locale" content="en_IE">
<meta property="og:site_name" content="Alan John">
<title>Home - Alan John</title>
<meta name="robots" content="index,follow">
<meta name="description" content="2022 Portfolio">
<meta property="og:title" content="Home - Alan John">
<meta property="og:description" content="2022 Portfolio">
<meta property="keywords" content="Alan, John, Software Developer, Sofware Engineer, Developer, Portfolio">
<meta property="og:url" content="https://alanjohn.dev/">
<meta property="og:image" content="https://alanjohn.dev/images/social_media_preview.png">
<link rel="canonical" href="https://alanjohn.dev/">

You can see how your site would appear in search results and social media posts at metatags.io.

robots.txt and sitemap.xml

The robots.txt file and sitemap.xml file are two files that are vital to how our page is crawled by google crawler bots. The robots.txt file lets good crawler bots know which routes they are allowed to crawl and index. You can add a robots.txt file to your NextJS site by adding it to your public directory.

# Allow all user agents.
User-agent: *
Allow: /

Sitemap: https://www.alanjohn.dev/sitemap.xml

The sitemap.xml is a file that lets the search engine crawler know what all routes are present on your website and how frequently they should be crawled and indexed. You could just manually add a sitemap.xml to your public directory but as there are dynamic paths that are created at build time, you would have to create a function to dynamically generate the sitemap.xml.

import { domain } from "./config";
import { BlogArticle } from "./interface";
import fs from "fs";

const sitemapxml = "public/sitemap.xml";

export const generateSiteMap = (posts: BlogArticle[]) => {
    const staticPages = fs
        .readdirSync("pages")
        .filter((staticPage) => {
            return ![
                "_app.tsx",
                "_document.tsx",
                "_error.tsx",
                "index.tsx",
            ].includes(staticPage);
        })
        .map((staticPagePath) => {
            return `${domain}/${staticPagePath.split(".")[0]}`;
        });

    staticPages.push(domain);

    const sitemap = `<?xml version="1.0" encoding="UTF-8"?>
    <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
      ${staticPages
          .map((url) => {
              return `
            <url>
              <loc>${url}</loc>
              <lastmod>${new Date().toISOString()}</lastmod>
              <changefreq>monthly</changefreq>
              <priority>1.0</priority>
            </url>
          `;
          })
          .join("")}
        ${posts
            .map((post) => {
                return `<url>
            <loc>${domain + "/blog/" + post.url}</loc>
            <lastmod>${post.modifiedDate}</lastmod>
            <changefreq>daily</changefreq>
            <priority>1.0</priority>
          </url>`;
            })
            .join("")}
    </urlset>
  `;

    fs.writeFileSync(sitemapxml, sitemap);
};

Social Media Preview Image

Whenever you share a link on whatsapp, telegram or any application, you get a preview which gives you a visual representation of the contents of that link. These social media preview images are important especially on social media sites like LinkedIn and Twitter to boost engagement. We are describing the social media image in the DefaultSEO and NextSEO components above. But the image we described is static in nature. It doesn’t change as per the page in the case of dynamic routing.

For regular static page routes, this might be tolerated. Though for blog page articles, not having a social media preview image that visually describes the content of the article might reduce the engagement. A descriptive social media image will go a long way into attracting people to click on the link.

To generate dynamic social media preview images for our blog articles, I have used two serverless functions implemented via Next.js API routes. I could have done the social media preview generation using one function but vercel has a 50mb size limit which gets exhausted by puppeteer-core and chrome-aws-lambda which I have utilized to create the social media preview image. So I created two API routes:

The /api/preview-html generates the preview using HTML, CSS and returns the HTML document in response.

import { BlogArticle } from "@util/interface"; // Type definition for blog
import { getBlogArticle } from "@util/notion"; // Gets blog article details
import { getHTML } from "@util/template"; // Creates HTML preview based on template

import type { NextApiRequest, NextApiResponse } from "next";

export default async function handler(
    req: NextApiRequest,
    res: NextApiResponse,
) {
    const { id } = req.query;
    try {
        const article: BlogArticle = await getBlogArticle(id.toString());
        const html = getHTML(article);
        res.setHeader("Content-Type", "text/html");
        res.end(html);
    } catch (error) {
        console.error(error);
        res.status(400).json(error);
    }
}

The /api/og-image opens the preview HTML in headless chrome instance with the help of puppeteer-core and chrome-aws-lambda. It takes a screenshot which it returns in response.

import { getPreviewHTML } from "@util/get-html-preview";
// Gets the preview-html url which we screenshot

import chrome from "chrome-aws-lambda";
import type { NextApiRequest, NextApiResponse } from "next";
import puppeteer from "puppeteer-core";

export default async function handler(
    req: NextApiRequest,
    res: NextApiResponse,
) {
    const { id } = req.query;
    try {
        const url = getPreviewHTML(id.toString());
        const options = process.env.AWS_REGION
            ? {
                  args: chrome.args,
                  executablePath: await chrome.executablePath,
                  headless: chrome.headless,
              }
            : {
                  args: [],
                  executablePath:
                      process.platform === "win32"
                          ? "C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe"
                          : process.platform === "linux"
                          ? "/usr/bin/google-chrome"
                          : "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
              };
        const browser = await puppeteer.launch(options);
        const page = await browser.newPage();
        await page.setViewport({ width: 1200, height: 628 });
        await page.goto(url, { waitUntil: "networkidle0" });
        const file = await page.screenshot({ type: "png" });
        res.statusCode = 200;
        res.setHeader("Content-Type", `image/png`);
        res.setHeader(
            "Cache-Control",
            `public, immutable, no-transform, s-maxage=31536000, max-age=31536000`,
        );
        res.end(file);
    } catch (error) {
        console.error(error);
        res.status(400).json(error);
    }
}

JSON-LD

We can help google search engine understand the content of our site even more by adding structured data with the help of JSON-LD. Google can use the structured data to enable special search features which in turn boost your page rank. You can read more about this here: https://developers.google.com/search/docs/advanced/structured-data/intro-structured-data

The Next-SEO plugin supports adding JSON-LD to our pages similar to how we add the meta tags. Though the ArticleJsonLd component did not pass the google rich text search. At the time of writing this article, it remains an unresolved issue. So I created my own CustomArticleJsonLd component based on how the ArticleJsonLd is designed.

interface Author {
    "@type": "person";
    name: string;
    url: string;
}

export interface CustomArticleJsonLdProps extends JsonLdProps {
    type?: "Article" | "BlogPosting";
    url: string;
    title: string;
    images: ReadonlyArray<string>;
    datePublished: string;
    dateModified?: string;
    authorName: Author | Author[];
    description: string;
}

export const CustomArticleJsonLd: React.FC<CustomArticleJsonLdProps> = ({
    type = "Article",
    keyOverride,
    url,
    title,
    images,
    datePublished,
    dateModified,
    authorName,
    description,
}: CustomArticleJsonLdProps) => {
    const data = {
        datePublished,
        description,
        mainEntityOfPage: {
            "@type": "WebPage",
            "@id": url,
        },
        headline: title,
        image: images,
        dateModified: dateModified || datePublished,
        author: authorName,
    };
    return (
        <JsonLd
            type={type}
            keyOverride={keyOverride}
            {...data}
            scriptKey="article"
        />
    );
};

export default CustomArticleJsonLd;

Using this component in my blog page route allowed me to add structured data which passed google’s rich text search.

<CustomArticleJsonLd
    type="BlogPosting"
    url={url}
    title={post.title}
    images={[socialimageurl]}
    datePublished={post.publishDate}
    dateModified={post.modifiedDate}
    authorName={{
        "@type": "person",
        name: "Alan John",
        url: "https://www.alanjohn.dev",
    }}
    description={truncated}
/>

What are web vitals?

💡 I have just summarised the core web vitals based on content found at https://web.dev/vitals/. My primary goal is to highlight how I improved my core web vitals and this is a prerequisite to that. If you want to learn more about web vitals in details, there is comprehensive content found at https://web.dev which i would strongly recommend. You can also refer NextJS learn.

Web vitals are a set of parameters introduced by google to quantify the user experience of a website. As of June 2021, Google has started using core web vitals in ranking pages. Core web vitals is a subset of web vitals that apply to all pages. As per web.dev, these are

Largest Contentful Paint (LCP): measures loading performance. To provide a good user experience, LCP should occur within 2.5 seconds of when the page first starts loading.
First Input Delay (FID): measures interactivity. To provide a good user experience, pages should have a FID of 100 milliseconds or less.
Cumulative Layout Shift (CLS): measures visual stability. To provide a good user experience, pages should maintain a CLS of 0.1. or less.

Improving this web vitals is not only beneficial to search engine rankings but also to judge the overall user experience. There is plenty of research online showing how improving web vitals has drastically improved business for multiple web based platforms. (Check out https://web.dev/vitals-business-impact/).

Measuring web vitals using Lighthouse

Lighthouse is a lab tool to measure web vitals in an isolated environment. You can use lighthouse online at https://web.dev/measure. You can use lighthouse locally by install the lighthouse npm package.

npm i -g lighthouse

You can also set up lighthouse-ci as a github action to evaluate the web vitals on push or in pull requests.

Beware that the vitals you get from lighthouse are from a controlled environment. This will slightly differ from your field data which is collected from your users. You can check out the differences in field data and lab data here. You can measure your site’s field data at pagespeed.web.dev. If you are deploying your site on vercel, it also provides a performance score based on web vitals collected from users.

Improving web vitals

I used the lab data that I got from lighthouse and worked towards improving those metrics. The web vitals that you get from lighthouse are the following

First Contentful Paint marks the time at which the first text or image is painted.
Speed Index shows how quickly the contents of a page are visibly populated.
Largest Contentful Paint marks the time at which the largest text or image is painted.
Time to interactive is the amount of time it takes for the page to become fully interactive.
Total Blocking Time is sum of all time periods between FCP and Time to Interactive, when task length exceeded 50ms, expressed in milliseconds.
Cumulative Layout Shift measures the movement of visible elements within the viewport.

Below are some of the things which I had to rethink and change to improve the web vitals.

Handling fonts

Typography is extremely important to the design of a website and the subsequent user experience. Though in the scope of this article, we would be more concerned with how fonts are loaded. Font loading affects the FCP and LCP. I was initially using fonts from Fontshare, Fontesk and Google fonts for my site. Google fonts has a lot servers and CDNs hosting them and NextJS provides webfont optimizations for fonts hosted on google fonts. The same can not be said about the fonts from Fontshare and Fontesk, I found it better to self host those fonts. Depending on how important the fonts are to a page, they had to be preloaded using the link tag. For me, my display font was vital to the home page as the text on that page was also the largest contentful paint. So I preloaded the display font for the / route.

Performance issues due to animations

I was using Framer motion and css animations in combination to create page transitions and micro-transitions. But framer motion isn’t a light-weight library and because I was utilizing it heavily, it led to a poorer FCP, TBT and Speed Index. So I removed framer motion and started to rely purely on CSS animations. Unfortunately that has it’s own limitations and I had to remove my page transitions altogether. Between web vitals and page transition animations, it was more practical to choose vitals. I am still looking at other javascript animation libraries and design patterns that could allow me to add those animations without sacrificing much performance. Open to suggestions.

Dynamic loading for code splitting

NextJS by default gives route based code splitting when we build the code. But that might not be sufficient. Component level code splitting can help reduce the unused javascript and lazy load certain components which are more sparingly used. The webpack bundle analyzer helped in analyzing in what my modules compose my chunks.

Previously when I was highlight.js, I was importing a lot of highlight.js modules which I wasn’t really using leading to inflation in size of chunks. So I removed a lot of them and later dynamically loaded them reducing the total size of all the chunks from 1.4mb to less than 500kb.

Highlight.js modules have been removed and dynamically loaded

💡 I had later removed highlight.js altogether for prism.js in this PR.

You have to be careful about not dynamically loading too many modules/components as it might increase your loading time.

React-Notion-X

In my previous article in this series, I had created my own react components to work with the Notion API due to the lack of a recognized renderer that works with Official Notion API. I came across the react-notion-X library but that did not use the Official Notion API then so I chose to create my own renderer. But soon after I published that article, Travis Fischer (creator of react-notion-x and a couple of really handy tools) added support for the official Notion API with notion-compat (still a WIP) which allows you to use react-notion-x’s powerful react renderer with the official API. So I replaced my renderer components with the react-notion-x renderer which gave me a huge performance boost.

NotionX / react-notion-x

Fast and accurate React renderer for Notion. TS batteries included. ⚡️

React Notion X

Fast and accurate React renderer for Notion. TS batteries included. ⚡️

Advice

If you just want to publish a website using Notion, then we highly recommend using Super.so — a hosted solution with great perf that takes care of all the details for you.

If you want more control over your website via React, then we recommend checking out the accompanying Next.js starter kit, which is free and uses react-notion-x under the hood.

And if you want even more control, then you're in the right place! 👇👇

Features

🚀 Simple - TypeScript + React
⚡ Fast - 10-100x faster than Notion
- 95-100% Lighthouse scores
- Heavier components can be loaded lazily via next/dynamic
💯 Tests - Comes with a comprehensive test suite covering most of Notion's functionality
🔥…

View on GitHub

💡 As far as why the renderer in react-notion-x outperformed what I had implemented for the scope of the project, the biggest difference in approach was that react-notion-x used react’s useMemo and useContext hooks followed by a lot of better design choices at block level.

Conclusion

There are still ways that I can improve the SEO and core web vitals. The objective of this series was not showcasing the right way to do this but to provide a case study for people trying to do something similar. I am open to suggestion on how I can improve the performance and user experience for the website even further but don’t have any further plans of documenting the process as part of this series.

You can find the source code here

alanpjohn / next-notion-portfolio

Portfolio cum Blog built with NextJS using Notion as a CMS

next-notion-portfolio

This project was build on a template generated with superplate Currently hosted on vercel at www.alanjohn.dev

Documented at

Available Scripts

Running the development server.

    npm run dev

Building for production.

    npm run build:analyze

Running the production server.

    npm run start

Linting the code

    npm run lint
    npm run lint:style

Learn More

To learn more about superplate, please check out the Documentation.

NextJS

A React framework for production with all the features you need for production: hybrid static & server rendering, TypeScript support, smart bundling, route pre-fetching, and more.

Go To Documentation

Notion API

Allows to connect to Notion databases giving you the power to use Notion as a CMS

Go To Documentation

TailwindCSS

…

View on GitHub

Creating a NextJS blog in typescript using Notion API

Alan P John — Sun, 27 Mar 2022 18:45:05 +0000

Prerequisites

This article is a follow up to my last article where I covered how to setup a NextJS repository for your developer portfolio. In this article, I would be covering how I used Notion as a headless CMS for my blog content.

It is expected that you know

How to create react components
how to use dynamic routing in NextJS
static site generation in NextJS with getStaticProps and getStaticPaths.
typescript

You can find the source code here.

Do you need a CMS?

In my last post, I explained how NextJS has MDX support and as developers we are used to writing in markdown. So for most developers, it might preferable to use MDX with their blog which would be a much simpler solution than integrating with a CMS. Then why did I chose to use Notion as a CMS? Primarily because I use Notion on a day to day basis to manage my study notes, work tracker, travel planner etc. So it made sense to store my blogs in Notion as well. There are some other benefits to use the Notion API as a headless CMS.

Having your code lie separately from your data gives you more flexibility. I can edit and manage my posts from the Notion website or the mobile app without having to make commits or pull requests. It makes for a cleaner repository where your commit history isn’t swamped with commits made to correct grammatical mistakes and updating content. The Notion application acts as a dashboard for me to manage my content and the website becomes the outlet where it is presented to the users. It also handles the issue of storing static assets as you can upload your pictures and videos to notion and then retrieve your content from there instead of putting all your static files in your /public directory.

Setting up Notion

The first you would need a Notion account. Sign up at notion.so and create your workspace. After that you would require a database on Notion for you to store and fetch blog articles from. You can duplicate the template I have used if you want to (this guide would follow this template). Just open the template and click on duplicate.

In the template that I made, I have the following columns

Title : title
Tags : multi_select
Publish : check_box
Date: last_edited_time
Description : rich_text
Link : url
PublishDate : Date

The title is the page.
The tags is a multi_select that allows us to add tags to our blog post.
Publish is a checkbox that controls whether this page is a draft or a published article on the site.
Date stores the last_edited_time to keep track of when an article was last updated.
The description is a rich_text summary.
Link is a url to another other site where the article was published.
PublishDate is the date on which it was published.

Now that you have a notion database to store your blog articles with a dummy article. Now you need to create an integration. For that, go to https://www.notion.com/my-integrations and create a new integration. Give it a name and give it read capabilities with the workspace you have created. Copy the token and save it somewhere securely. Next go to your database and click on share. Now you add your integration name here and copy the database ID.

https://www.notion.so/a8aec43384f447ed84390e8e42c2e089?v=...
                      |--------- Database ID --------|

Store your notion integration token and the ID of your database in the .env file inside your directory. Do not push this file to github.

NOTION_KEY=<YOUR NOTION INTEGRATION TOKEN>
NOTION_BLOG_DATABASE_ID=<YOUR NOTION BLOG DATABASE ID>

You are now all set. Follow the getting started documentation provided by notion for more details.

Retrieving data from Notion API

Go to your NextJS directory and install the notion client.

$ npm install @notionhq/client

💡 There are a lot of unofficial open source notion integration libraries out there you should check them out to see if they meet your requirements or not before making your own integration. I might separate the code that I wrote to integrate notion to create React components into a separate library for others to use in the near future.

Let’s look at the sample code of the two API calls we’ll be using. The first is to query the database to collect all blog article data.

import { Client } from "@notionhq/client";
const notion = new Client({ auth: process.env.NOTION_KEY });

export async function getBlogPosts(){
    const databaseId = process.env.NOTION_BLOG_DATABASE_ID;
    const response: QueryDatabaseResponse = await notion.databases.query({
        database_id: databaseId,
    });

    return response;
}

Retrieving the page data is slightly different. Notion stores its page data in the form of blocks. So to get the content of a page, you need to retrieve the blocks. Here is how you would retrieve the blocks in a page.

export const getBlocks = async (id: string) => {
    let response = await notion.blocks.children.list({
        block_id: id
    });

    return response;
};

When you retrieve the blocks for a page, you only get one level of blocks. So you’ll have to send subsequent requests for every block to retrieve any children of the block if the block has any children.

Typing Issues

If you’re not using typescript then the you don’t have to worry about the typing issue.

When using the notion API in typescript, you’ll find it difficult to use the typing provided as notion auto-generates the typing which leads to a large union of types aggregated in a few types. This poses a problem when you want a type definition for a specific property or block type. You don’t have a type defined for them as those definitions are part of very large union (which isn’t easily readable either). This is not ergonomic to work with. You can learn more about this issue here.

You could just use the any type but that isn’t a recommended practice. A better workaround would be to use the extract utility type. The extract type is a generic type will help us extract the specific type we need from a union of types.

Getting all blog posts from Notion database

Lets look at our blog database query response. If you print the query database response on the console, you would get something like this.

{
  object: 'list',
  results: [
    {
      object: 'page',
      id: '270434234-31fc-4193-86e2-5ebd7f0de8de',
      created_time: '2022-02-18T18:27:00.000Z',
      last_edited_time: '2022-03-25T17:44:00.000Z',
      created_by: [Object],
      last_edited_by: [Object],
      cover: [Object],
      icon: null,
      parent: [Object],
      archived: false,
      properties: [Object],
      url: 'https://www.notion.so/TestPage-270bd3023413c419386e25ebd7f0de8de'
    }
  ],
  next_cursor: null,
  has_more: false,
  type: 'page',
  page: {}
}

The results member of the QueryDatabaseResponse object holds the database entries. The database entry would consist of a properties object which holds the data stored in each column of your database table.

If you look at the type defintion of the response.results on your IDE in the tooltip, you would see that it is a very large union of type definitions. Similarly, the type definition for response.results.[*].properties is an even bigger union of type definitions. Using extract, we can get the exact type definition we need from the union and give it an alias. Having these aliases will allow you to safely extract information from your query database response and store them in a object which you can use more easily.

import {
    QueryDatabaseResponse,
} from "@notionhq/client/build/src/api-endpoints";

export type PostResult = Extract<
    QueryDatabaseResponse["results"][number],
    { properties: Record<string, unknown> }
>;

Now, PostResult type is an alias to the type definitions in response.results with a properties attribute. We can then extract the type definitions for the specific property types used in our columns using extract as well.

import {
    QueryDatabaseResponse,
} from "@notionhq/client/build/src/api-endpoints";

export type PostResult = Extract<
    QueryDatabaseResponse["results"][number],
    { properties: Record<string, unknown> }
>;

type PropertyValueMap = PostResult["properties"];
type PropertyValue = PropertyValueMap[string];

type PropertyValueType = PropertyValue["type"];

type ExtractedPropertyValue<TType extends PropertyValueType> = Extract<
    PropertyValue,
    { type: TType }
>;

export type PropertyValueTitle = ExtractedPropertyValue<"title">;
export type PropertyValueRichText = ExtractedPropertyValue<"rich_text">;
export type PropertyValueMultiSelect = ExtractedPropertyValue<"multi_select">;
export type PropertyValueUrl = ExtractedPropertyValue<"url">;
export type PropertyValueDate = ExtractedPropertyValue<"date">;
export type PropertyValueEditedTime =
    ExtractedPropertyValue<"last_edited_time">;

Now let’s define an interface for our post data which we would require.

export interface IPost {
    id: string;
    url: string;
    tags: string[];
    modifiedDate: string;
    publishDate: string;
    title: string;
    description: string;
    link?: string;
}

Now, we’ll extract an array of IPost from the QueryDatabaseResponse.

type DatabaseItem = PostResult & {
    properties: {
        Title: PropertyValueTitle;
        Date: PropertyValueEditedTime;
        Tags: PropertyValueMultiSelect;
        Description: PropertyValueRichText;
        Link: PropertyValueUrl;
        PublishDate: PropertyValueDate;
        LastUpdated?: PropertyValueDate;
    };
};

const extractPosts = async (
    response: QueryDatabaseResponse,
): Promise<IPost[]> => {
    const databaseItems: DatabaseItem[] = response.results.map(
        (databaseItem) => databaseItem as DatabaseItem,
    );
    const posts: IPost[] = await Promise.all(
        databaseItems.map(async (postInDB: DatabaseItem) => {
            const title = postInDB.properties.Title.title[0].plain_text;
            const date = postInDB.properties.Date.last_edited_time;
            const description =
                postInDB.properties.Description.rich_text[0].plain_text;
            const url = getCanonicalURL(title);
            const link = postInDB.properties.Link.url || "";
            const tags = postInDB.properties.Tags.multi_select;
            const cover = await getPageCover(postInDB.id);
            const publishdate = postInDB.properties.PublishDate.date?.start;

            const post: IPost = {
                id: postInDB.id,
                title: title,
                modifiedDate: date,
                description: description,
                url: url,
                link: link,
                cover: cover,
                tags: tags,
                publishDate: publishdate || date,
            };
            return post;
        }),
    );
    return posts;
};

export async function getBlogPosts(): Promise<IPost[]> {
    const databaseId = process.env.NOTION_BLOG_DATABASE_ID || "";
    const response: QueryDatabaseResponse = await notion.databases.query({
        database_id: databaseId,
    });
    console.log(response);
    const posts = await extractPosts(response);
    return posts;
}

The property types we created previously using extract help us in get the information we require from the QueryDatabaseResponse without having to deal with possible undefined fields. Now, the getBlogPosts function returns an array of IPost which is much easier to work with.

The getCanonicalURL function creates a URL for the blog post based on its title.

export const getCanonicalURL = (title: string): string => {
    const cleaned = title.replace(/\W/gm, " ");
    const removedSpaces = cleaned
        .split(" ")
        .filter((str) => str)
        .join("-");
    return removedSpaces;
};

Getting all blocks of a page

Now that we have the ID of all our blog pages. We can retrieve the blocks for each page. Let’s look at the ListBlockChildrenResponse that we get when we retrieve the blocks.

{
  object: 'list',
  results: [
    {
      object: 'block',
      id: 'a6fc6649-1a48-4be7-9772-f945780b09fe',
      created_time: '2022-02-19T08:11:00.000Z',
      last_edited_time: '2022-03-25T17:41:00.000Z',
      created_by: [Object],
      last_edited_by: [Object],
      has_children: false,
      archived: false,
      type: 'bookmark',
      bookmark: [Object]
    },
    ... // Truncated
    {
      object: 'block',
      id: '191d3863-cd7b-45ca-8b82-83c968b5be3a',
      created_time: '2022-03-25T17:44:00.000Z',
      last_edited_time: '2022-03-25T17:44:00.000Z',
      created_by: [Object],
      last_edited_by: [Object],
      has_children: false,
      archived: false,
      type: 'paragraph',
      paragraph: [Object]
    }
  ],
  next_cursor: null,
  has_more: false,
  type: 'block',
  block: {}
}

You only get one level of blocks when you retrieve the blocks of a page. If one block has child blocks, you’ll have to call the function again with the block ID to get it’s children. You can know if a block has children by seeing the value of has_children .
Depending on the block type, the object will have different member. For "paragraph” type blocks, the information about the block is stored in paragraph member and so on for all the block types offered by Notion. The type definitions for these are again not properly defined as everything inside ListBlockChildrenResponse is defined as a union of type definitions.

So to properly extract information from the blocks, we’ll again use the Extract utility class to extract the block type definitions.

export type Block = Extract<
    ListBlockChildrenResponse["results"][number],
    { type: string }
>;

export type BlockType = Block["type"];

type ExtractedBlockType<TType extends BlockType> = Extract<
    Block,
    { type: TType }
>;

export type ParagraphBlock = ExtractedBlockType<"paragraph">;

export type HeadingOneBlock = ExtractedBlockType<"heading_1">;
export type HeadingTwoBlock = ExtractedBlockType<"heading_2">;
export type HeadingThreeBlock = ExtractedBlockType<"heading_3">;

export type HeadingBlock =
    | HeadingOneBlock
    | HeadingTwoBlock
    | HeadingThreeBlock;

export type BulletedListItemBlock = ExtractedBlockType<"bulleted_list_item">;
export type NumberedListItemBlock = ExtractedBlockType<"numbered_list_item">;

export type QuoteBlock = ExtractedBlockType<"quote">;
export type EquationBlock = ExtractedBlockType<"equation">;
export type CodeBlock = ExtractedBlockType<"code">;
export type CalloutBlock = ExtractedBlockType<"callout">;
export type ToggleBlock = ExtractedBlockType<"toggle">;
export type EmbedBlock = ExtractedBlockType<"embed">;
export type WebBookmarkBlock = ExtractedBlockType<"bookmark">;
export type ImageBlock = ExtractedBlockType<"image">;

Notion uses the same definition for rich text and file objects so we can create aliases for that as well for reusability.

export type RichText = ParagraphBlock["paragraph"]["rich_text"][number];
export type File = ImageBlock["image"];

As we have seen when we printed ListBlockChildrenResponse, the Block type that we have extracted doesn’t have an attribute to store children in it. But it would be better for us if we can store the children of block inside the block object itself. So we define a new type that extends the extracted Block type.

export type BlockWithChildren = Block & {
          type: BlockType;
          childblocks: BlockWithChildren[];
      }

Now to retrieve all the blocks inside the page.

export const getBlocks = async (blockId: string): Promise<Block[]> => {
    const blocks: Block[] = [];
    let response = await notion.blocks.children.list({
        block_id: blockId,
    });

    response.results.map((block) => {
        blocks.push(block as Block);
    });

    return blocks;
};

The max number of blocks you can get per request is 100, so you’ll have to utilize pagination to get all the blocks if they exceed a 100.

export const getBlocks = async (blockId: string): Promise<Block[]> => {
    const blocks: Block[] = [];
    let response = await notion.blocks.children.list({
        block_id: blockId,
        page_size: 25,
    });

    response.results.map((block) => {
        blocks.push(block as Block);
    });
    while (response.has_more && response.next_cursor) {
        response = await notion.blocks.children.list({
            block_id: blockId,
            page_size: 25,
            start_cursor: response.next_cursor,
        });
        response.results.map((block) => {
            blocks.push(block as Block);
        });
    }
    return blocks;
};

Now we also need a function to get the children of the block if the block has children and convert the Block object into a BlockWithChildren object.

const getChildren = async (block: Block): Promise<BlockWithChildren> => {
    const children: BlockWithChildren[] = [];
    if (block.has_children) {
        const childBlocks = await getBlocks(block.id);
        const childBlocksWithChildren = await Promise.all(
            childBlocks.map(async (block) => await getChildren(block)),
        );
        childBlocksWithChildren.map((block: BlockWithChildren) => {
            children.push(block);
        });
    }
    const ablock: BlockWithChildren = {
        ...block,
        childblocks: children,
    };
    return ablock;
};

The getChildren method takes a Block and recursively retrieves the children for the block if it has any and returns a BlockWithChildren. Now adding all of it together, I have created a getPageBlocks method which will return an array of BlockWithChildren having all the blocks of the page.

export const getBlocks = async (blockId: string): Promise<Block[]> => {
    const blocks: Block[] = [];
    let response = await notion.blocks.children.list({
        block_id: blockId,
        page_size: 25,
    });

    response.results.map((block) => {
        blocks.push(block as Block);
    });
    while (response.has_more && response.next_cursor) {
        response = await notion.blocks.children.list({
            block_id: blockId,
            page_size: 25,
            start_cursor: response.next_cursor,
        });
        response.results.map((block) => {
            blocks.push(block as Block);
        });
    }
    return blocks;
};

const getChildren = async (block: Block): Promise<BlockWithChildren> => {
    const children: BlockWithChildren[] = [];
    if (block.has_children) {
        const childBlocks = await getBlocks(block.id);
        const childBlocksWithChildren = await Promise.all(
            childBlocks.map(async (block) => await getChildren(block)),
        );
        childBlocksWithChildren.map((block: BlockWithChildren) => {
            children.push(block);
        });
    }
    const ablock: BlockWithChildren = {
        ...block,
        childblocks: children,
    };
    return ablock;
};

export const getPostBlocks = async (
    pageId: string,
): Promise<BlockWithChildren[]> => {
    const blocks: Block[] = await getBlocks(pageId);
    const blocksWithChildren: BlockWithChildren[] = await Promise.all(
        blocks.map(async (block: Block) => {
            const blockWithChildren = await getChildren(block);
            return blockWithChildren;
        }),
    );
    return blocksWithChildren;
};

The getBlogPosts function and the getPageBlocks function should be called in the getStaticProps method of your page. The page will built at runtime so you don’t have to worry about your site making repeated requests to your notion API each time the user requests the page. With ISR, you can make sure your pages are up to date with the content inside Notion by rebuilding the pages after a certain time period.

Rendering Page Content

Now that we have an array of BlockWithChildren, we can just iterate through the array and return a react component based on the type of block. We can similarly render the children of the block inside that react component.

const renderBlock = (block: BlockWithChildren): React.ReactNode => {
    const childblocks: BlockWithChildren[] = block.has_children
        ? block.childblocks
        : [];
    const content: React.ReactNode = childblocks.map(
        (block: BlockWithChildren) => {
            return renderBlock(block);
        },
    );
    switch (block.type) {
        case "paragraph":
            return <Paragraph key={block.id} {...block} />;
        case "heading_1":
            return <Heading1 key={block.id} {...block} />;
                /* Truncated code for readability */
        default:
                        // to handle unsupported block by our integration
            return <NotSupportedBlock key={block.id} reason={block.type} />;
    }
};

export type PostContentProps = {
    blocks: Array<BlockWithChildren>;
};

export const PostContent: React.FC<PostContentProps> = ({
    blocks,
}: PostContentProps) => {
    return (
        <article>
            {blocks.map((block: BlockWithChildren) => {
                return renderBlock(block);
            })}
        </article>
    );
};

And then inside our page, we can use the PostContent component.

<PostContent blocks={blocks} />

Now let’s look at how we handle the common blocks.

Text Blocks

When I mean text blocks, I am referring to paragraph, headings, callouts and quotes. These blocks have rich text objects inside them which are presented in different ways on the frontend. So all we have to do is make a function to render the rich text and present them inside the react components we make for these blocks. If you look at the type definitions for these block types, you’ll notice they have an array of RichText stored in the rich_text member. We’ll take this array and return a span for each RichText. The text content of a RichText object is stored in the plain_text member. RichText can be bold, italic, code, strikethrough, underlined, links, different colors etc so we’ll have to add that in the styling of the span.

export const renderText = (
    id: string,
    textBlocks?: Array<RichText>,
): React.ReactNode => {
    if (!textBlocks) {
        return <></>;
    }
    let count = 0;
    return textBlocks.map(({ annotations, plain_text, href }) => {
        const { bold, code, color, italic, strikethrough, underline } =
            annotations;
        count = count + 1;
        return (
            <span
                key={`text-${id}-${count}`}
                className={[
                    bold ? "bold" : "",
                    code ? "mono" : "",
                    italic ? "italic" : "",
                    strikethrough ? "strikethrough" : "",
                    underline ? "underline" : "",
                ].join(" ")}
                style={color !== "default" ? { color } : {}}
            >
                {href ? (
                    <a className="default-link not-prose" href={href}>
                        {plain_text}
                    </a>
                ) : (
                    plain_text
                )}
            </span>
        );
    });
};

Based on that, the react component for paragraph type blocks would look like

type ParagraphBlockProps = PropsWithRef<ParagraphBlock>;

export const Paragraph: React.FC<ParagraphBlockProps> = ({
    id,
    paragraph,
}: ParagraphBlockProps) => {
    return <p>{renderText(id, paragraph.rich_text)}</p>;
};

List Blocks

List blocks are more complicated to handle as Notion treats lists similar to how markdown handles lists. They do not follow a nested structure.

- Item 1
    - SubItem 1
    - SubItem 2
- Item 2
    - SubItem 3
        - SubItem4

Meanwhile in HTML, this would be represented differently

<ul>
    <li> 
        Item 1
        <ul>
            <li>
                SubItem 1
            </li>
            <li>
                SubItem 2
            </li>
        </ul>
    </li>
    <li>
        Item 2
        <ul>
            <li>
                SubItem 3
                <ul>
                    <li>
                        SubItem 4
                    </li>
                </ul>
            </li>
        </ul>
    </li>
</ul>

In HTML, the list items needs to be nested inside a <ul> or <ol> tag. When we get the bulleted_list_item or the ordered_list_item type of block, they don’t have any data indicating whether they belong to the same list or not. So we need to pre-process the list items that we get from Notion to create the nested structure of lists. My approach has been to create my own ListBlock type which I extend the extracted BlockWithChildren type definition.

export type ListBlock = {
    id: string;
    object: string;
    type: "bulleted_list" | "numbered_list";
    childblocks: BlockWithChildren[];
    has_children: boolean;
    archived: boolean;
    created_time: string;
    last_edited_time: string;
};

export type ListItemBlock = {
    id: string;
    object: string;
    type: "list_item";
    childblocks: BlockWithChildren[];
    has_children: boolean;
    archived: boolean;
    list_item: BulletedListItemBlock["bulleted_list_item"];
    created_time: string;
    last_edited_time: string;
};

export type BlockWithChildren =
    | (Block & {
          type: BlockType;
          childblocks: BlockWithChildren[];
      })
    | ListBlock
    | ListItemBlock;

The new ListBlock allows me to create a nested structure where I put adjacent bulleted_list_item or ordered_list_item types of block into a ListBlock object and put the contents of these list item blocks into ListItemBlock objects. So the ListBlock represents my ul and ol tags while the ListItemBlock represents my li tag. I have used queues to convert all the bulleted_list_item or ordered_list_item types of blocks into a ListBlock oject with an array of ListItemBlock objects as its children.

const createListBlock = (
    blocktype: "bulleted_list" | "numbered_list",
    blocks: Array<BlockWithChildren>,
) => {
    const processedChildren: BlockWithChildren[] = blocks.map(
        (block: BlockWithChildren) => {
            if (
                block.type == "bulleted_list_item" ||
                block.type == "numbered_list_item"
            ) {
                const blockContent =
                    block.type == "bulleted_list_item"
                        ? block.bulleted_list_item
                        : block.numbered_list_item;
                const ablock: ListItemBlock = {
                    ...block,
                    type: "list_item",
                    list_item: blockContent,
                };
                return ablock;
            }
            return block;
        },
    );
    const block: BlockWithChildren = {
        object: blocks[0].object,
        id: blocks[0].id,
        created_time: new Date(Date.now()).toISOString(),
        last_edited_time: new Date(Date.now()).toISOString(),
        has_children: true,
        archived: false,
        type: blocktype,
        childblocks: processedChildren,
    };
    return block;
};

export const extractListItems = (
    blocks: Array<BlockWithChildren>,
): Array<BlockWithChildren> => {
    const postprocessed = Array<BlockWithChildren>();
    const bulleted_list_stack = Array<BlockWithChildren>();
    const numbered_list_stack = Array<BlockWithChildren>();

    blocks.forEach((block: BlockWithChildren) => {
        switch (block.type) {
            case "bulleted_list_item":
                bulleted_list_stack.push(block);
                break;
            case "numbered_list_item":
                numbered_list_stack.push(block);
                break;
            default:
                if (bulleted_list_stack.length > 0) {
                    postprocessed.push(
                        createListBlock("bulleted_list", bulleted_list_stack),
                    );
                } else if (numbered_list_stack.length > 0) {
                    postprocessed.push(
                        createListBlock("numbered_list", numbered_list_stack),
                    );
                }
                postprocessed.push(block);
                bulleted_list_stack.length = 0;
                numbered_list_stack.length = 0;
                break;
        }
    });

    if (bulleted_list_stack.length > 0) {
        postprocessed.push(
            createListBlock("bulleted_list", bulleted_list_stack),
        );
    } else if (numbered_list_stack.length > 0) {
        postprocessed.push(
            createListBlock("numbered_list", numbered_list_stack),
        );
    }

    return postprocessed;
};

The extractListItems function takes the Array of BlockWithChildren which doesn't have a nested list structure and returns the Array of BlockWithChildren with the ListBlock objects. We need to call this function to pre-process any array of type BlockWithChildren before we create react components for it.

const renderBlock = (block: BlockWithChildren): React.ReactNode => {
    const childblocks: BlockWithChildren[] = block.has_children
        ? extractListItems(block.childblocks) // Preprocessing list items
        : [];
    const content: React.ReactNode = childblocks.map(
        (block: BlockWithChildren) => {
            return renderBlock(block);
        },
    );
    switch (block.type) {
        case "paragraph":
            return <Paragraph key={block.id} {...block} />;
        case "heading_1":
            return <Heading1 key={block.id} {...block} />;
                /* Truncated code for readability */
        default:
            return <NotSupportedBlock key={block.id} reason={block.type} />;
    }
};

export type PostContentProps = {
    blocks: Array<BlockWithChildren>;
};

export const PostContent: React.FC<PostContentProps> = ({
    blocks,
}: PostContentProps) => {
        const blocksWithList = extractListItems(blocks); // Preprocessing list items
    return (
        <article>
            {blocksWithList.map((block: BlockWithChildren) => {
                return renderBlock(block);
            })}
        </article>
    );
};

The react components for List blocks would be as follows.

type ListBlockProps = PropsWithChildren<ListBlock>;

export const UnorderedList: React.FC<ListBlockProps> = ({
    children,
}: ListBlockProps) => {
    return <ul>{children}</ul>;
};

export const OrderedList: React.FC<ListBlockProps> = ({
    children,
}: ListBlockProps) => {
    return <ol>{children}</ol>;
};

type ListItemBlockProps = PropsWithChildren<ListItemBlock>;

export const ListItem: React.FC<ListItemBlockProps> = ({
    id,
    list_item,
    children,
}: ListItemBlockProps) => {
    return (
        <li>
            {renderText(id, list_item.rich_text)}
            {children}
        </li>
    );
};

Code Blocks

Code blocks have a extra layer of complexity over text blocks which is syntax highlighting. We will use highlight.js for syntax highlighting. First, we install highlight.js.

$ npm i highlight.js

In your _app.js , add your preferred highlight.js stylesheet. You can see a full list of highlight.js stylesheets here.

import "highlight.js/styles/github-dark-dimmed.css";

highlight.js contains support for a lot of languages, most of which you won’t be needing. Importing syntax highlighting for all the languages will cause your site to load slower. Even the common languages subset is very big. I would recommend creating another file where you configure your highlight.js instance.

import { HLJSApi } from "highlight.js";
import hljs from "highlight.js/lib/core";
import bash from "highlight.js/lib/languages/bash";
import c from "highlight.js/lib/languages/c";
import cplusplus from "highlight.js/lib/languages/cpp";
// add remove languages as per your preference

export const getConfiguredHighlight = (): HLJSApi => {
        // register the languages
    hljs.registerLanguage("bash", bash);
    hljs.registerLanguage("shell", shell);
    hljs.registerLanguage("c", c);
    hljs.registerLanguage("cplus", cplusplus);

        // add aliases for flexibilty
        hljs.registerAliases(["c++", "cplusplus"], { languageName: "cplus" });

    hljs.configure({ ignoreUnescapedHTML: true });

    return hljs;
};

Now to highlight the code syntax inside the react component for code blocks, we import the configured hljs and highlight the code element.

import { renderText } from "@components/notion/text";

import { getConfiguredHighlight } from "@util/highlight";
import { CodeBlock } from "@util/interface";

import { PropsWithRef, useEffect, useRef } from "react";

type CodeBlockProps = PropsWithRef<CodeBlock>;

export const MultilineCodeBlock: React.FC<CodeBlockProps> = ({
    id,
    code,
}: CodeBlockProps) => {
    const ref = useRef<HTMLElement>(null);

    useEffect(() => {
        const hljs = getConfiguredHighlight();
        if (ref.current) {
            hljs.highlightElement(ref.current);
        }
    });

    return (
        <pre className="bg-codeblock">
            <code ref={ref} className={`${code.language}`}>
                {renderText(id, code.rich_text)}
            </code>
        </pre>
    );
};

Image Blocks

NextJS provides in built-image optimization with it’s next/image component. You will have to specify the domains from where the images are fetched in your NextJS configuration. It is easy to add the domains for whenever you upload an image to notion. But it is not feasible to handle images which aren’t uploaded to notion. So for now till we find a workaround for that, we’ll avoid the external image case. You can check where your upload images are stored and add the domain name to your next.config.js.

module.exports = {
    images: {
        domains: [
            "s3.us-west-2.amazonaws.com",
        ],
    },
});

A problem you would come across with the next/image component is displaying responsize images without knowing the size of the image beforehand. We can solve that using the fill layout option and css styling.

type ImageProps = PropsWithRef<ImageBlock>;

export const BlogImage: React.FC<ImageProps> = ({ id, image }: ImageProps) => {
    const altText = image.caption
        ? image.caption.map((richText) => richText.plain_text).join(" ")
        : "Some image";
    const src = image.type == "file" ? image.file.url : "external";
    const children = renderText(id, image.caption);
    if (src == "external") {
        return (
            <NotSupportedBlock
                key={id}
                reason={`Image type ${image.type} not supported`}
            />
        );
    }
    return (
        <figure className="blog__image">
            <Image src={src} layout="fill" className="image" alt={altText} />
            {children && <figcaption>{children}</figcaption>}
        </figure>
    );
};

.blog__image {
        width: 100%;
        position: relative;

        > div,
        span {
            position: unset !important;
        }

        .image {
            object-fit: contain;
            width: 100% !important;
            position: relative !important;
            height: unset !important;
        }
    }

Please note I have used SCSS, the CSS code snippet for this will be slightly different.

What’s next?

You can create react components for other blocks like embed helping you create a more rich user experience.
You can generate your non blog pages like a on-site resume or details about your projects etc from Notion as well. (I have done that so you can refer that in the source code).
You can use dynamic loading to improve the performance of your site.

Building a Developer Portfolio: Setting up my NextJS repository with the help of Superplate

Alan P John — Sun, 13 Mar 2022 19:31:18 +0000

A developer portfolio is a good place to centralize all the things about you as a software developer, from you projects and open source contributions to networking through social platforms. It allows you to showcase who you are and what you are capable of. And this does not necessarily apply just to front developers or freelance developers.

Like any project, a developer portfolio should be well thought of before starting. What do you want to showcase? How do you want it to look? How do you want to manage it? what tools and frameworks to use? In this article, I am going to take you through the tools I have used and why I chose to use them. It’s important to keep in mind the “why”, as it’s easy to lose track of what is required even in something as simple as a static portfolio site. This article is the first part of a series where I break down my developer portfolio whose code can be found on github.

Selecting NextJS

I have used NextJS as it’s an excellent react framework to make production ready single page applications. NextJS is an excellent choice for static sites like blogs and portfolios with some of the features it offers.

Static Site Generation with Incremental Static Rendering

The most important feature that NextJS provides which makes it a great choice for developing sites like blogs is static site generation (SSG) with incremental static rendering (ISR). Static site generation in NextJS happens in the absense of getInitialProps() and getServerSideProps(). NextJS pre-renders the page at build time that can be cached in CDN making delivery faster and improving SEO.

export const getStaticProps: GetStaticProps<BlockPageProps> = async () => {
    const posts = await getBlogPosts(); // Gets posts at build time
    return {
        props: {
            posts: posts, // the data is passed to page as part of props
        },
                revalidate: 3600,
    };
};

export default BlogPage;

Next.js allows you to create or update static pages after you’ve built your site. Incremental Static Regeneration (ISR) enables you to use static-generation on a per-page basis, without needing to rebuild the entire site. This happens with the help of revalidate attribute. Lets say you set revalidate at 60 seconds. The page is pre-rendered at build time and the static file is served at request for the 60 second interval. The first request after the 60s interval would trigger a regeneration for that static page. Once the new page is generated, it will replace the old static page in the cache. This is useful in blogs as we don’t have to manually build the whole web app again whenever there is a change in the blog database.

How vercel handles NextJS ISR

MDX and AMP support

Markdown is a developer’s preferred way of documentation. So it’s natural many developers would prefer storing their blogs posts and content in the form of markdown and have their site generator convert their markdown into websites. NextJS supports MDX which allows you embed your react component into markdown allowing you to make feature rich markdown sites.

MDX support in NextJS

NextJS also supports converting your react pages into AMP pages which provides a rich experience to reading articles and boosts your search engine rankings and user engagement.

Image and font optimizations

Images and fonts are important static assets for any website and key to their design. But often they come with their own set of problems when it comes loading and serving them to different viewports. NextJS provides an Image component to handle all the problems you could face with your images. Images are rendered such that there is no cumulative layout shift. Images are lazy loaded by default but that can be changed to prioritize.

Image Optimization in Next.js

Accessing fonts from third party application can slow down your load time. NextJS provides in-built font optimization for fonts provided by Google Fonts and Adobe Fonts (previously Typekit).

Font Optimization in Next.js

It is easy to start with NextJS as there is a large community around it plus the docs are comprehensive and easy to understand even for advanced use cases. Though Gatsby is another framework you might want to look at. Especially when it comes to developing static sites, Gatsby has a slight edge over NextJS in performance. At the day, both are amazing frameworks for your applications, so check them out if you haven’t already.

Setting up your site

I used SuperplateCLI to generate a boilerplate closest to my requirements. Superplate is a library that allows you to create a customizable boiler plate code for react and nextjs sites. There is work going to add more frameworks. You can use Superplate and create a boilerplate with your preferred libraries and tools.

$ npx superplate-cli test

✔ Cloned remote source successfully.
✔ Select your project type › nextjs
✔ What will be the name of your app · test
✔ Package manager: · npm
✔ UI framework: · tailwind
✔ CSS Preprocessor: · scss
✔ Features: · env,bundle-analyzer
✔ Hooks · 
✔ State Management: · none
✔ i18n - Internationalization · none
✔ Linting tools: · eslint,prettier
✔ Do you want to use lint-staged? · none
✔ Testing Framework: · none
✔ E2E Testing framework: · none
✔ Docker integration: · none
✔ Continuous integration: · github-actions

Success! Created test at /tmp/test

You can run several commands:

  npm run dev
    Starts the development server.

  npm run build
    Bundles the app for production.

  npm run start
    Starts the production server.

Start developing by typing:

  cd test
  npm run dev

It can seem a little overwhelming for beginners when you use superplate to know which options you want to select. Superplate has very good documentation breaking down what each option offers but also how to set them up later manually. Here are some of the tools and libraries that I generally include.

💡 There are a lot more plugins that Superplate offers. I am just covering a handful that I am using for my portfolio.

Tailwind css with SCSS

Tailwind CSS is a handy CSS framework especially if you are someone who hates having ridiculously large css files in your repositories. Tailwind is a utility first css framework which allows you to compose your styling within your HTML with the help of predefined classes. It gives you more control compared to other frameworks as there is no predefined theme or style. When you build tailwind, it only includes the css for the classes that you have used. Tailwind allows you to create your own classes extending tailwind for reusability and customization. It makes writing media queries much easier for responsive design and has support for dark mode. Personally tailwindcss has made my work more readable and easier to manage but it might not be to everyone’s way of working. The tailwind typography plugin is especially useful fo styling blog components.

More about Tailwind CSS

Eslint and Prettier

ESlint is a JS linting tool which checks your code for styling and syntax errors. It helps maintain higher standards of code quality. Prettier examines your files for style issues and reformats your code automatically to guarantee that consistent standards for indentation, space, semicolons, and single vs double quotes are followed. Using both can increase developer productivity by identifying problems. Personally, even though initially might feel like it’s a hassle but as time goes you find yourself writing better cleaner code. Both offer a variety of plugins and options allowing you to decide your style. If you are using eslint with NextJS, you should check out the @next/eslint-plugin-next which ensures you are using the best practices with NextJS and making the most of it. Similarly prettier has a plugin for tailwind which can organise your utility classes better.

💡 Use the --fix flag with ESlint to allow prettier and eslint to fix issues automatically wherever possible

Bundle analyzer

Bundle analyzer visualises the sizes of various modules included in your bundle in an interactive zoom map. It is useful in analyzing your bundles and debugging them.

More about bundle analyzer

Github actions

It’s 2022 and you should be using a CI to automate builds and test commits and pull requests. As this is hosted on github, I’ll be using Github actions to test and check the build for commits on master. I’ll also create a action to run lighthouse but I’ll cover that in more detail in a future article where I cover core web vitals.

Next Steps

Once you generate you superplate boilerplate, you might want to check all the scripts and run npm audit to check for security vulnerabilities before you start work. Some more libraries that you can include which were not part of superplate can be:

Framer Motion: Good websites employ animations and transitions to give their users a smooth experience. Framer motion is a declarative motion library for react.
Next-SEO: SEO is an important part of building a website and NextSEO simplifies SEO for next sites.
Stylelint : While ESLint is a linting tool for your JS code, stylelint is tool for linting your css and scss code.

Deep-dive into Containerization: Creating containers from scratch

Alan P John — Sat, 05 Mar 2022 19:55:09 +0000

Containers have changed the landscape for how we design, develop, and deploy your applications. Today, cloud native technologies are transforming IT ecosystems largely thanks to containerization. In this article, I’ll be creating a container runtime using shell commands. Ideally it’s not recommended to implement your own container runtime. This article is just to get a better understanding of lower level linux functionalities which will help in

Designing more secure images.
Using images more efficiently.
Debugging while using higher level tools

Linux Fundamentals

Before I start the deep dive, it is important to be familiar with certain linux concepts.

Cgroups

Cgroups limit the resources, such as memory, CPU, and network input/output, that a group of processes can use. There is a hierarchy of control groups for each type of resource being managed, and each hierarchy is managed by a cgroup controller. Any Linux process is a member of one cgroup of each type, and when it is first created, a process inherits the cgroups of its parent.

The Linux kernel communicates information about cgroups through a set of pseudo-filesystems that typically reside at /sys/fs/cgroup. You can see the different types of cgroups on your system by listing the contents of that directory.

user@myPChostname:~$ ls /sys/fs/cgroup/
blkio  cpuacct      cpuset   freezer  memory  net_cls           net_prio    pids  systemd
cpu    cpu,cpuacct  devices  hugetlb  misc    net_cls,net_prio  perf_event  rdma  unified

If you have docker installed on the system and look inside the /sys/fs/cgroup/memory directory, you’ll find a directory for docker. All the files in this directory define different kinds of memory limits on your docker containers. You’ll find a similar directory in /sys/fs/cgroup/cpu where the cpu limits for your docker containers are defined.

Namespaces

By putting a process in a namespace, you can restrict the resources that are visible to that process.

Linux kernel 5.6 currently provides 8 namespaces:

pid : provides a process with its own set of process IDs
net : allows processes to have their own network stack
mnt : abstracts filesystem view and manages mount points
ipc : provides separation of named shared memory segments
user : provides processes with their own set of user IDs and group IDs
uts : allows processes to have own domain name and hostname
cgroup : allows a process to have its own set of cgroup root directories
time : virtualize the clock of the system

A process is always in exactly one namespace of each type. When you start a Linux system it has a single namespace of each type. You can easily see the namespaces on your machine using the lsns command.

The unshare command allows us to create sub processes that don't share namespaces with their parent process. You can also use the nsenter command to specify namespaces for a process. In this article, I’ll stick to using unshare.

Creating a Containerized Process

Containers seem to be very similar to virtual machines, but it’s crucial to understand that they are very different. While virtual machines emulate a complete machine, including the operating system and a kernel, containers share the kernel of the host machine and, as explained, are only isolated processes.

Hostname

Let’s start by isolating the hostname. If you run the hostname command from within a docker container, you can see that it’s a different hostname than your host.

user@myPChostname:~$ hostname
myPChostname
user@myPChostname:~$ docker run --rm -it --name hello centos bash
[root@f1e54241a12b /]$ hostname
f1e54241a12b

To achieve a similar isolation, we need to give its own UTS namespace using the unshare command.

💡 I am running these bash commands on an ubuntu VM created using multipass.

To create a new UTS namespace, we can use the --uts flag with unshare.

ubuntu@host:/$ hostname
host
ubuntu@host:/$ sudo unshare --uts bash
root@host:/$ hostname child
root@host:/$ hostname
child

If you were to open another terminal window to the same host before exit, you can confirm that the hostname hasn’t changed for the whole (virtual) machine.

ubuntu@host:/$ hostname
host

Filesystem

Next, we need to give our containerized process it’s own root filesystem so that it does not access the host root. We’ll be using the --root option to do that. This will help us assign a directory as the new root. But before we do that, for any directory to be a root directory, it requires a root filesystem which includes directories such as /bin, /proc etc. So I am going to download the alpine minirootfs to quickly create a minimal root filesystem in my new directory. You can also export root filesystems from existing docker containers if you want.

ubuntu@host:~$ mkdir container_root
ubuntu@host:~$ cd container_root/
ubuntu@host:~/container_root$ curl -o alpine.tar.gz https://dl-cdn.alpinelinux.org/alpine/latest-stable/releases/x86_64/alpine-minirootfs-3.15.0-x86_64.tar.gz
ubuntu@host:~/container_root$ tar xvf alpine.tar.gz
ubuntu@host:~/container_root$ rm alpine.tar.gz
ubuntu@host:~/container_root$ ls
bin  etc   lib    mnt  proc  run   srv  tmp  var
dev  home  media  opt  root  sbin  sys  usr

So now to use the --root option with the unshare command

ubuntu@host:/$ sudo unshare --uts \
--root=/home/ubuntu/container_root \
sh
/$ pwd
/
/$ ls /
bin    etc    lib    mnt    proc   run    srv    tmp    var
dev    home   media  opt    root   sbin   sys    usr

The root directory of the containerized process is no longer the root directory of our host system. Which also means we can’t use the commands like bash as the /bin in our new root filesystem doesn’t have bash even though the host machine /bin did.

ubuntu@host:/$ sudo unshare --uts \
--root=/home/ubuntu/container_root \
bash
chroot: failed to run command ‘bash’: No such file or directory

Processes

Now if we run the ps command, as you can see we can’t see any processes at all. That’s because the ps command runs by listing the /proc psuedo-filesystem. All processes have their own directory within the /proc . You can run ls /proc on your linux system to see how it looks like. You can read more about the /proc filesystem here.

You can mount /proc using the mount command or by using the --mount-proc flag.

ubuntu@host:/$ sudo unshare --uts  \
--mount-proc=proc \
--root=/home/ubuntu/container_root \
sh
/$ ps
PID   USER     TIME  COMMAND
    1 root      0:01 {systemd} /sbin/init
    2 root      0:00 [kthreadd]
    3 root      0:00 [rcu_gp]
    4 root      0:00 [rcu_par_gp]
    6 root      0:00 [kworker/0:0H-kb]
        ... <truncated>

Now, we can see all the processes running on host which is not right. Containers should not be able to access the processes of the host machine. To isolate the host processes, We use the --pid flag with unshare to get a new PID namespace. Along with that we also need to use the --fork flag. This is useful when creating a new pid namespace as --fork runs the specified program as a child process of unshare rather than running it directly.

ubuntu@host:/$ sudo unshare --uts \
--pid --fork \
--mount-proc=proc \
--root=/home/ubuntu/container_root sh
/$ ps
PID   USER     TIME  COMMAND
    1 root      0:00 sh
    2 root      0:00 ps

Mounts

Now we have our processes isolated. The next namespace we need to look into is the mount namespace. We can do that using the --mount flag in the unshare command. This isolation comes handy in keeping in making sure host directories mounted into container are not visible from other containers.

ubuntu@host:~$ sudo unshare --uts \
--pid --fork \
--mount \
--mount-proc=proc \
--root=/home/ubuntu/container_root \
sh

Networking interfaces

Containers have their own networking interface and routing tables. This requires the process to have a separate network namespace which can be set using --net flag.

ubuntu@host:~$ sudo unshare --uts \
--net \
--pid --fork \
--mount \
--mount-proc=proc \
--root=/home/ubuntu/container_root \
sh
/$ ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

When we create our network namespace, we only have a loopback interface. The container will be unable to communicate if it only has a loopback interface. We need to estabilish a virtual Ethernet interface connecting the container network namespace to the default one.

While keeping the containerized process running in one terminal. Open another with root priveleges. Create a virtual ethernet interface on your host machine. You’ll need to know your container’s pid for that. We can use the lsns command to find that.

ubuntu@host:/$ sudo lsns -t net
        NS TYPE NPROCS   PID USER    NETNSID NSFS COMMAND
4026531992 net      93     1 root unassigned      /sbin/init
4026532193 net       2  2241 root unassigned      unshare --uts --net --pid --fork --mount --mo
ubuntu@host:/$ sudo ip link add ve1 netns 2241 type veth peer name ve2 netns 1

Then we need to get the connection up. On the host machine

ubuntu@host:/$ sudo ip link set ve2 up

In the container process

/$ sudo ip link set ve1 up

Now that the connection is up, we assign IP. on the host machine run

ubuntu@host:/$ sudo ip addr add 192.168.1.200/24 dev ve2

on the container process run

/$ ip addr add 192.168.1.100/24 dev ve1

now you should be able to ping the host from the container and vice versa, allowing your container to communicate with other processes.

Inter process communication

Different processes communicate with each other with the help of a shared range of memory. For that, they need to part of the same IPC namespace.We generally wouldn’t want our containers to be able to access one another’s shared memory. In which case we can use the --ipc flag.

ubuntu@host:~$ sudo unshare --uts \
--net --ipc \
--pid --fork \
--mount \
--mount-proc=proc \
--root=/home/ubuntu/container_root \
sh

Cgroups

You can use the --cgroup flag to create a new cgroup namespace which makes sure that your container process cannot see any higher cgroup configuration.

ubuntu@host:~$ sudo unshare --uts \
--net --ipc --cgroup \
--pid --fork \
--mount \
--mount-proc=proc \
--root=/home/ubuntu/container_root \
sh

Users

Currently, the user in the containerized process is the root user because we use sudo.

ubuntu@host:~$ sudo unshare --uts \
--net --ipc \
--pid --fork \
--cgroup --mount \
--mount-proc=proc \
--root=/home/ubuntu/container_root \
sh
/ $ id
uid=0(root) gid=0(root) groups=0(root)

To prevent this, we create a separate user namespace for the container process with the help of --user flag.

ubuntu@host:~$ sudo unshare --user \
--uts --net --ipc \
--pid --fork \
--cgroup --mount \
--mount-proc=proc \
--root=/home/ubuntu/container_root \
sh
~ $ id
uid=65534(nobody) gid=65534(nobody) groups=65534(nobody)

Now the user assigned to the container process is "nobody". We can map this uid to a non-root user on the host machine by making a change in the /proc/<pid>/uid_map where <pid> is your container process pid. The user namespace is created first when you run unshare with the --user flag and you are automatically root in the container user namespace. This means you can create namespaces inside the containerized process while running unshare without sudo allowing us to run containers without any root privileges. (rootless containers)

ubuntu@host:~$ unshare --uts \
--net --ipc \
--pid --fork \
--cgroup --mount \
--mount-proc=proc \
--root=/home/ubuntu/container_root \
sh
unshare: unshare failed: Operation not permitted
ubuntu@host:~$ unshare --user \
--uts --net --ipc \
--pid --fork \
--cgroup --mount \
--mount-proc=proc \
--root=/home/ubuntu/container_root \
sh
~ $

And here we have our container ready!!

All your container runtime tools are wrappers around these in-built features which provide you more ease and flexibility of configuration.