DEV Community: Stephan Meijer

Linking Packages with Yalc

Stephan Meijer — Sat, 08 Jul 2023 10:13:14 +0000

As web developers, we often work on projects involving multiple packages or modules, that are under simultaneous development, making it necessary to link them together to test integration. The traditional approach for this uses npm link, which creates a symbolic link between the package in development and the global node_modules directory. The better way is to use yalc.

What is Yalc?

Yalc is a simple package manager for local package development. It acts as a simple local repository for your packages that you can publish to and install from, just like you would with a remote package registry, such as npm or yarn. Yalc simplifies package linking by avoiding some of the complexities and pitfalls associated with npm link, yet it's much easier to work with than running a local registry like verdaccio. Yalc is a set of scripts, not a running service.

Why use Yalc over npm link?

npm link creates a symbolic link for your local package in the global node_modules directory. This can lead to confusion as packages can unknowingly share dependencies, causing unintended side effects. If you've ever tried to link a package that depends on react, the "invalid hook call" error might sound familiar.

Yalc operates more like a local publishing system. Instead of linking, it pushes the package content into the consumer project, mimicking the process of publishing and installing a package. This results in an experience closer to working with an actual dependency, minimizing environment inconsistencies and unexpected behavior.

How to use Yalc?

Using Yalc is straightforward. First, you need to install it globally on your machine:

npm install -g yalc

After it's installed, navigate to the directory of the package. Then make sure to build the package and run yalc publish. Yalc only copies the files over that should be published to NPM, so building is essential.

npm run build
yalc publish

After running that command, your package has been "published" to the global Yalc store at ~/.yalc/packages. Then, in your consumer project, you can install it as you would with a regular npm package:

yalc add <your-package-name>

That does a couple of things. It updates the yalc.lock at your project, copies the package over from the global store, to the local one at .yalc, and modifies package.json to add the dependency or update it to point to the files in the .yalc/<your-package-name> directory.

Your package is now added to your project as if it were a regular dependency. When you change your package and want to update it in your project, you can rebuild it, and use yalc push to push an update. yalc push will publish the package to the global store, and push the changes to all linked projects local stores in one command.

yalc push

For the best experience, you want to hook up yalc push to your build --watch command, so that it automatically pushes changes to your linked projects. But manual pushing goes a long way too.

Retreat and Restore

As I mentioned earlier; yalc doesn't create symlinks, it copies the publishable assets of the package into your project. Your project will get a .yalc directory, containing the package, and a yalc.lock file. Both can be committed to your repository, but if you should or not is up to you. I don't, but I can imagine some use cases where you might need to.

If you don't commit them you'll have to run yalc retreat --all before you commit. That command removes all yalc references from the package.json. Complementary, yalc restore reads your yalc.lock and restores any missing links.

Final words

While npm link offers a quick way to link local packages, Yalc provides a more robust and predictable workflow that mimics using a real dependency. Using Yalc, you can avoid some of the pitfalls of npm link and make your local package development more efficient and reliable.

Liked this article? Share your thoughts on Twitter, and check my other articles.

Isomorphic Development

Stephan Meijer — Sat, 08 Jul 2023 10:11:43 +0000

At first, I wanted to name this post "No, Server Components Don't Make Your Job Easier", but that wouldn't have covered it. It's not server components itself complicating our jobs. It's the isomorphic nature of it. The fact that the same code runs in both server and client, and that they're supposed to bridge the gap between them. Supposed to, because we're not there yet.

Sure, server components will make your react components look simpler. That's because server components come with strict regulations. Don't use hooks, or client state. Don't make them interactive. Guess how complex your client side react components would look, if you'd follow the same rules? Exactly, they'd look simpler too.

Every time someone tries to sell me the idea that moving code to the server makes our lives easier, they fail to explain how. With server components some go as far as saying that not seeing how they simplify stuff, "is the same resistance we showed against JSX years ago", and that "we just need to adapt our mental model". I can tell you, it's not. Moving anything from client to server, is nothing like moving from HTML or createElement to JSX.

I've been writing isomorphic code since Meteor.js was released back in 2012. So I want to be clear that I'm not against fullstack development, and I'm not even against isomorphic development, as I fully embrace its benefits. What I am against, is the false claim that moving parts of our code to the server, would make the job of a frontend developer easier.

Isomorphic Development: Bridging The Network

Let's clear out the first misconception. Frameworks like Next.js and Remix let you write server and client code neatly together, and you'll boot your environment with a single npm run dev command. That doesn't mean that it's one environment tho.

There's still a server, and still a client/browser bundle. In Next the server is code generated by extracting methods like getServerSideProps or the new functions decorated with 'use server' directives , and in Remix it's functions like loader and action. Server code runs on the server, Client code runs in the browser, but depending on your implementation it might also run on the server. Let that last one sink in for a moment. That's what this writing is about.

The thing to understand is that the build tools separate your server and client logic, and run each in their own environment. Server code on the server, client code in the browser, and sometimes, but not always, client code on the server. That, is what makes it not easier.

With this paradigm shift, we're not asking frontend developers to render their code on the server, we're asking them to write code that's runs safely, and performant, in both environments.

Debugging: Juggling Two Environments

With code split between the frontend and backend, you're suddenly managing two separate environments. It means watching two debuggers hitting breakpoints, each with its own context. Or for the console.loggers amongst us, you'll now have log statements in two consoles.

Also remember that your component might render twice. Once in the context of the server, and once in the context of the client. Might, because Next.js took a different direction than Remix. In Remix, your component will render twice. Once on the server, and then on the client during hydration (attach js handlers to make interactive). Next.js chose the route of not supporting state/interactivity on server components, so no hydration/rerender is needed there. Or at least, not for the server components, client components still need hydration.

Security: Protecting Boundaries

Fullstack developers must guard against data leaks and breaches. The server has access to sensitive information that clients shouldn't have. You'll now have to think about how to get those properties safely to the client.

Did you once store a variable on the module scope (out of your react component) as a form of cache between renders? Move that component from 'use client' to 'use server' during a refactor, and you'll leak information between users.

It might not be hard to fix, but it's an easy bug to introduce when your build tool decides where to run the code. Something to keep in mind and be aware of. At any time.

Performance: The Dual Nature

Performance behaves differently on the client and server. On the client, speed matters and we avoid browser bottlenecks. To use our time efficient, we often work under the principle "you don't run the function a thousand times, so fast is fast enough". Well, move that function from the client to the server, and scalability takes center stage. Functions that once ran only a couple of times, now run a couple times on every user request.

And remember, Node.js is single threaded. All sync code that runs on the server, block other requests being made. In the browser you don't notice a 10ms delay, on your server, it can cause requests to queue up and your AWS bill to grow.

Data Querying: Bridging The Gap

Querying data is a joint effort. Frontend make API calls, while backend tackles databases. Fullstack developers must master database performance, query optimizations, and indexing. Don't forget about rate limiting challenges when server interacts with external APIs. Where the client would have made a handful requests to an API from thousands of different origins, you're now making thousands requests from a single origin.

When caching responses for the sake of performance, be aware of where your query runs, for the sake of security. Depending on the query, you might or might not want to share your caches between requests.

Final words

My concern isn't about moving to fullstack development. I've been there for over 15 years. My concern is about the hybrid nature that we're moving into. Code runs in both environments, and boundaries are there, but not clearly visible. It's easy to slip up because we're busy, or because we're missing something during refactor.

Our tools aren't ready for this. We don't have an isomorphic debugger that hits breakpoints on both environments, and we don't have an isomorphic console that merges log statements from client and server. Eslint is merely a linter for formatting, and TypeScript won't warn you about leaking variables either. Last time I've checked, we even don't have a code coverage tool merging results from both sides.

And all of the above can be learned. But it's so much more than "just adapt your mental model". It's new knowledge, it's more knowledge, and it'll make your job harder. If you believe it doesn't, then you're either missing the risks, or you're underselling yourself.

While we're at it, maybe it's time to stop calling it fullstack development. It's no longer enough to be proficient in both environments. We need to start thinking of it like a single environment, even tho our tools don't expose it that way. How does isomorphic development sound?

Liked this article? Share your thoughts on Twitter, and check my other articles.

Don't mutate the iterator

Stephan Meijer — Sat, 08 Jul 2023 10:11:37 +0000

Or better said, don't mutate the iterator you're iterating over. This is a common mistake that can lead to unexpected results. And yes, despite writing about mutating and not mutating before, I still got bitten by it. So let's take a look at what happened.

The problem

I needed to remove all empty headers before a request. For this, I created a function that mutates the request.headers object. Let's take a look:

function deleteEmptyHeaders(headers: Headers) {
  headers.forEach((value, key) => {
    if (value && value !== 'undefined' && value !== 'null') return;
    headers.delete(key);
  });
}

This function iterates over the headers, and removes the ones that are "empty". Usage would be like:

const headers = new Headers();
headers.set('a', '');
headers.set('b', '');
headers.set('c', '');
headers.set('d', '');

deleteEmptyHeaders(headers);
Object.fromEntries(headers); // { b: '', d: '' } // ?!?

The problem is that we're mutating the iterator we're iterating over. By removing the first header, the second header shifts up to the first position. Like when you're pulling a book from a pile of books. As the iterator is now in the new second position, it never processes "header b" but instead processes "header c" as the new second header. And so on.

It's an old problem that we know of iterating over indexes using a for loop, like for (let i = 0; i < items.length; i++). When we remove an item from the array, the indexes shift, and skip items. The same thing happens here. It's just harder to spot.

The solution

So how do we fix this? Well, we can't mutate the iterator we're iterating over. So we need to create a new iterator. We can do this by using the Array.from method.

function deleteEmptyHeaders(headers: Headers) {
  const entries = Array.from(headers.entries());
  entries.forEach(([key, value]) => {
    if (value && value !== 'undefined' && value !== 'null') return;
    headers.delete(key);
  })
}

Instead of Array.from we can also use the spread operator [...headers.entries()], and calling the .entries() method is optional as it's the default method to be called on such actions. So let's apply this:

function deleteEmptyHeaders(headers: Headers) {
  [...headers].forEach(([key, value]) => {
    if (value && value !== 'undefined' && value !== 'null') return;
    headers.delete(key);
  })
}

Now we don't modify the iterator we're iterating over, and we end up with the expected result. Or said differently, the index of the current header isn't affected by the delete action, as we read a different pile than we modify.

Please do notice that the key and value props got reversed when we moved from headers.forEach to array.forEach.

Bonus

Twitter being Twitter, I got a lot of feedback on my snippet. From folks being unable to spot the bug, to folks saying that a return in a forEach is bad practice, to completely burning me down. So let's take a look at some of the feedback.

folks that write code like this, get what they deserve

Yeah, thanks for the constructive feedback. Now go away. I didn't want to quote this one at first, but it's important to highlight that this is not the way to give feedback. It's not constructive, and it's not helpful. It's just rude.

don't mutate

A couple of folks suggested that I shouldn't mutate. Well, I don't have another option. I need to modify the request headers before the request is made. And request.headers is a read-only property. It is what it is. Tho I agree that generally speaking, modifying the input arguments is not good practice. As always, there are exceptions to the rule.

The other option would be to not set the header in the first place. If you have that option, go with it. At other times, cleanup is the only option we have.

don't return in forEach

I don't get it. The callback is a function. We can use early returns to reduce indentation and keep code readable. I don't see the problem here. Some folks suggested using continue or break instead, but that's not going to work. It's a function; we can't use those keywords. Returning from a forEach to exit early is fine. Note that it won't stop the iteration, but it will stop the execution of the callback.

key and value are reversed

Yeah, I know. It is the function signature tho. Seriously, why did TC39 decide to put the value first? It's so confusing.

const headers = new headers();

// value, key
headers.forEach((value, key) => {});

// key, value
[...headers].forEach(([key, value]) => {});

you're checking 'null' and 'undefined' instead of null and undefined

Yep. headers.set coerces the value into a string. header.set('key', null) will result in headers.get('key') === 'null'.

use a common loop

Sure, possible. This is the direct translation, including the same bug. Using forEach or a traditional loop is about personal preference. Switching between them doesn't change a thing.

function deleteEmptyHeaders(headers: Headers) {
  for (const [key, value] of headers) {
    if (value && value !== 'undefined' && value !== 'null') continue;
    headers.delete(key);
  }
}

Interestingly enough, this makes the bug easier to spot for me. It's hard to say if that's because I'm used to this pattern or because I now know it's there.

Conclusion

Just a couple of lines and so many opinions and lessons. The "common for loop" could have been suggested in a kinder tone, but seeing this implementation in a traditional loop does remind me why I so strongly prefer those over helper methods like map and forEach.

This is the final solution I came up with and the way I've implemented it in fetch-addons.

function deleteEmptyHeaders(headers: Headers) {
  for (const [key, value] of [...headers]) {
    if (value && value !== 'undefined' && value !== 'null') continue;
    headers.delete(key);
  }
}

request.headers.set('x-api-key', apiKey);
request.headers.set('x-user-id', userId); // string | null | undefined

// somewhere in a middleware
deleteEmptyHeaders(request.headers);

Liked this article? Share your thoughts on Twitter, and check my other articles.

Simpler Dev Environments with Procfiles

Stephan Meijer — Wed, 01 Jun 2022 19:50:55 +0000

Throughout the time, I've tried a number of different ways to manage my development environment. From Make files, to PM2 and Tmux.

Tmux, is well.., tmux. Tmuxinator makes it manageable, but the "layout" definition is far from readable, and copying errors from history to your clipboard, is a true pain in the ass.

I still recommend taking a look at PM2 if you need more than "get things running". It comes with a lot of options, but that also means that your config script comes with some verbosity.

This article explains how I cover my simple use cases. Those that don't require additional file watchers (see PM2), or split panes to separate logs while sticking to a single terminal (see Tmux). For those, I use Procfiles.

No fear, Procfile is here!

Procfile? Yeah, the first time I heard it I was "not another make file, right?!". And luckily, it's nothing like that. Let's get started.

So a Procfile is a simple key: command format. And much like a Dockerfile, it's by convention named after the format. I recommend creating file named Procfile in the root of your project, and commit it to your repo. If not for yourself, than for every new contributor that one day joins your team.

For MagicBell, our procfile looks as follows:

server: bin/rails s -p 3000 -e development
worker: bin/bundle exec sidekiq
webpack: bin/webpacker-dev-server

That's it. That's all the config you need to get multiple services up and running through a single command.

Obviously, we still need to install a runner to handle this procfile. Meet foreman, or one of it's forks. Foreman is a Ruby script, so for that you'll need to have Ruby installed. There are many forks though. Such as shoreman, which is a dependency free shell script, or node-foreman, which is a javascript fork. I go with node-foreman, for the simple reason that I'm a node guy and I like that I can npm install it to the dependencies of my node projects.

So, go ahead, and install node-foreman. For the sake of this howto, let's go global. Drop the -g if you have a concrete project at hand.

npm i -g node-foreman

Node-foreman installs a binary named nf. With this installed, it's as simple as running nf start in the same directory as your Procfile. Try it, and see all your services spin up and log to the same terminal. When you have an .env file in the same directory, node-foreman automatically loads all environment variables from it. If your env file is named differently, say .env.development, you can specify it with the --env flag. Use the --procfile flag if you need to specify a Procfile.

A bit more… tabs

The above might be all you need. But sometimes, I like to have my startup process a bit personalised. For that, I use scripts that I store under my home directory.

For MagicBell, my config exists of two files. One custom procfile, saved as ~/scripts/magicbell.proc:

server: bin/rails s -p 3000 -e development
worker: bin/bundle exec sidekiq
webpack: bin/webpacker-dev-server
codegen: yarn codegen -w

And the "executable" script that I run, saved as ~/scripts/start-magicbell (don't forget to chmod +x)

#!/usr/bin/env bash

cd ~/dev/magicbell/backend
ttab docker-compose up

bundle install
yarn install
yarn db:migrate

nf start -j ~/scripts/magicbell.proc -e .env.development

When read that script, you'll notice ttab. ttab is a small utility that allows us to start commands in a new terminal tab. That way, I can start postgres & redis in one terminal instance (tab) while running everything else in another. Ttab is optional, but I highly recommend using it for simplicity.

npm i -g ttab

As I've added ~/scripts to my PATH, I can get the MagicBell environment up and running using a single start-magicbell command. Let me walk you trough wat it does:

first, it navigates to the git repo on my drive, at ~/dev/magicbell/backend
runs docker-compose up in a new tab, detached from the current
installs all ruby gems, while the db is spinning up
installs all node modules
runs our database migration script
starts the four services defined in the procfile

With this single command, I've started 6 services and ran some install/migrate commands. It happened too often that I manually had to install new gems or modules after pulling changes. Those commands are fast enough to run as part of my startup, yet take enough time for the database to be online before the server starts.

That's it. Both node-foreman and ttab come with a bunch of options. So be sure to check out their readmes (especially from node-foreman!)

👋 I'm Stephan. Follow me on Twitter if stuff like this has your interest.

Conflicted

Stephan Meijer — Mon, 28 Mar 2022 21:32:42 +0000

A LOT has happened over the last few months. And yet, I haven't written about any of it. I wanted to, but I've been unable to find the time, the words, or maybe mostly the right mood.

I wanted to write about my adventure with Kent C. Dodds. We've created one of the first, if not the first, full-fletched website using Remix. It was lots of fun. Little did I know that the animations and transitions that I implemented for Kent would lead to my current position at MagicBell.

Together with the launch of his new website, my "Chat with Kent" got published. Again, something worth writing about. I remember thinking back then that I should have said things differently. But I still haven't listened to the recording.

The launch of his site lead to a chat with Hana Mohan. Not long after, I signed for a role as Founding Frontend Developer at MagicBell, and ended my partnership at a firm that I co-founded but got never recognized for. I've sold my 10% and moved on.

As the cherry on the pie, we had a company retreat in Miami. It was a fantastic way to get to know my new coworkers. I'm truly grateful for that experience. That trip alone is worth writing for.

Since I started working for MagicBell, I've added several features to the product, the dashboard, the SDKs, and launched MagicBell Playground. A little side project under business hours.

But then the conflicted part.

While working on Kent's new website, my step-dad got diagnosed with cancer. He is unlikely to make the end of this year. Doctors have given him months, and he's running out of time.

While I quit my partnership and transitioned to my new position at MagicBell, my wife booked a last-minute flight "home," as her grandma was hospitalised with Covid, and doctors gave her a 50/50 chance of survival. Luckily she did.

Only to be abruptly awakened by incoming Russian artillery on February 24th. Two days after my 35th birthday. While I launched the playground, grandma hid in the bathtub. She's here now. Safe with us. But it took us two weeks to get her out of the country.

We managed to extract six more friends. Or, well, one friend with her two kids and their grandma and one aunt with her grandson. Other friends fled on time or are still stuck in this dirty war. Some because they don't want to leave their home. Others because they simply can't.

As I'm writing this, grandma is crying again. She received a photo of her apartment earlier today, and it doesn't look nice.

The last few months have been an emotional rollercoaster. My professional life has been nothing less than amazing. Sure, there are some frustrations, but that's nothing. On a personal level, I do my best to be happy and positive for the kids, while I'm heartbroken for the lives of our friends.

In a few days from now, my kid turns 2. He'll get his first bike, and we'll eat pie. And then again, one of the guests will be our friend with her kids, who lost their home, miss their dad, and wear our kids' clothes.

I'm grateful. I'm happy. I'm terrified. I wish I could do more.

Typed GraphQL with react-query & graphql-request

Stephan Meijer — Thu, 25 Nov 2021 10:19:13 +0000

While our front-end at MagicBell is a React/TypeScript application, we write our back-end in Ruby. This article explains how I've connected our client to the back-end and how we guard against breaking API changes.

GraphQL Code Generator

Let's get the obvious choice out of the way first. graphql-codegen. It's a no-brainer. This tool enables us to generate TypeScript types based on a GraphQL schema. Think typed queries, mutations, fragments, and object types.

Having types generated based upon your GraphQL schema means that TypeScript will inform you when the back-end (GraphQL API) introduces a breaking change. As a front-end engineer, that's what I want!

GraphQL Code Generator can generate fully typed React hooks if you tell it to, but I'm a fan of keeping things simple and thereby of their TypedDocumentNode approach. This variant is unaware of the GraphQL client that you're using. In other words, it's not tied to react-apollo (or alternative).

To get that up and running, you'll need to install a few dev dependencies:

npm i -D 
  @graphql-codegen/cli 
  @graphql-codegen/typed-document-node
  @graphql-codegen/typescript 
  @graphql-codegen/typescript-operations 
  @graphql-typed-document-node/core

And now the more exciting part, our codegen config (codegen.yml). We don't have a schema stored in code, and because the back-end is in Ruby, graphql-codegen won't be able to extract the types out of the back-end source files either. So instead, we provide our GraphQL endpoint. It's an easy setup, but the one downside is that our server must be running when we want to generate new types.

schema:
  - http://localhost:3000/graphql:
      headers:
        X-MAGICBELL-API-KEY: "${MAGICBELL_API_KEY}"
documents: "./app/javascript/src/graphql/**/*.graphql"
generates:
  ./app/javascript/src/graphql/generated.ts:
    plugins:
      - typescript
      - typescript-operations
      - typed-document-node

That's it. Codegen will read the .env file from the project root and use that to populate the API headers. documents is the path where the GraphQL queries are stored. I'll come back to those later.

Graphql Client

I'm not going with the famous @apollo/client because it's too much for our dashboard. Besides, we're migrating from REST to GraphQL, meaning we'll need to deal with both for some time. As React Query is sublime in cache management, and can be used for both REST and GraphQL, we'll be using that.

I have considered swr instead of react-query, as it's smaller, but they lack some fundamentals that we need. Think clear state indicators or even a (solid) solution to manage mutations.

For the fetching part, we'll use graphql-request. It has almost as many installs/month as apollo, but it's way smaller and doesn't have as many open issues.

TypedDocumentNode and graphql-request

Now the "tricky" part, we need graphql-request to use our generated TypedDocumentNode. For that, I've created a custom hook:

import { useCallback } from 'react';
import { request } from 'graphql-request';
import { RequestDocument } from 'graphql-request/dist/types';
import { TypedDocumentNode } from '@graphql-typed-document-node/core';
import { useCurrentUser } from '../context';

export function useGraphqlRequest() {
  const { apiKey } = useCurrentUser();

  return useCallback(
    <TDocument = any, TVariables = Record<string, any>>(
      document: RequestDocument | TypedDocumentNode<TDocument, TVariables>,
      variables?: TVariables,
    ) =>
      request<TDocument, TVariables>('/graphql', document, variables, {
        'X-MAGICBELL-API-KEY': apiKey,
      }),
    [apiKey],
  );
}

That's the magic sauce that will power it all. It returns a graphql-request client and uses TypedDocumentNode to infer types from the query. It also provides some defaults to graphql-request. Even without the type annotations, this would be a helpful hook that prevents setting options (like the header) in multiple places throughout our code.

Writing queries

And with this setup, we have a way to query the back-end with confidence. To create a new query, I'll write a GraphQL definition in a *.graphql file somewhere in the path declared by codegen.yml#documents.

For example, this logs query:

query logMessage($id: ID!) {
  log(id: $id) {
    id
    createdAt
    user {
      firstName
      lastName
      email
    }
    notification {
      title
      content
      actionUrl
    }
  }
}

Then run npx graphql-codegen, and consume the generated types in my query hook that I compose using react-query:

import { useQuery } from 'react-query';
import { useGraphqlRequest } from './useGraphqlRequest';
import { LogMessageDocument, LogMessageQuery } from './generated';

type UseLogMessageOptions = {
  logId?: LogMessageQuery['log']['id'];
};

export function useLogMessage({ logId }: UseLogMessageOptions) {
  const request = useGraphqlRequest();

  return useQuery<LogMessageQuery['log']>(
    ['log-message', logId],
    () => request(LogMessageDocument, { id: logId }).then((x) => x.log),
    { enabled: logId != null },
  );
}

Lastly, we'll consume that hook in our component, to fetch the data that we need:

function LogDetails({ logId }: LogDetailsProps) {
  const { data, status } = useLogMessage({ logId });

And that's it. data is fully typed. Suppose the back-end engineers introduce a change that isn't compatible with the current front-end, or I'm consuming data that never existed in the first place. In that case, TypeScript will notify us by throwing errors as part of the checks that run against our pull requests.

Recap

We type our GraphQL queries using graphql-codegen and use react-query to manage server/query state. graphql-request is the glue between codegen and react-query, a typed fetch for GraphQL, if you want. And with this setup, I have reduced the chance of breaking our GraphQL queries at MagicBell.

Handling POST requests in Next.js getServerSideProps

Stephan Meijer — Fri, 10 Sep 2021 08:11:56 +0000

The documentation pages of Next.js list getServerSideProps under "data fetching", and for any kind of data mutations, they'll point you to the api routes. I'm here to tell you, that there is another way!

In this article, I'm going to show you how you can post to getServerSideProps, and have your components, data fetching, and data mutation logic all collocated in the same file.

HTML Forms

I'll directly dive in. We'll create a form so we can bring this to a working example. Let's keep it simple. Create a new next app with npx create-next-app, and add a form to your index component.

export default function IndexPage({ name = "person" }) {
  return (
    <form method="post">
      <input name="name" defaultValue={name} />
      <button type="submit">submit</button>
    </form>
  );
}

Note that I don't set the "action" attribute. When the action attribute is left out, it defaults to the current path. And that's exactly what we want it to do. If you'd be using an hosted form service like rake.red, you'd be setting it to an absolute url.

As we'll be adding getServerSideProps in the next step, I've also already provided an initial name value via the props, and default it to person for now.

That's it for the component part. This will render a form that accepts a name, and will be submitted to the current path. Give it a spin!

Server Side Props

So let's start with the basic variant of getServerSideProps.

export async function getServerSideProps({ req }) {
  console.log(req.method, req.body);

  return {
    props: {
      name: "smeijer"
    }
  };
}

I've added an initial value for name, which you'll now see rendered in your form. The value no longer comes from the component, but it's consumed from an "backend part". See it as a query if you will, but without additional request, and without a need for react-query, swr, or other forms of client state management.

Now, if you refresh the page, and resubmit the form, you'll notice that the server will log GET on refresh, and POST on form submit. That's right! We don't need anything special to receive the post. Next already does that out of the box.

But here's the thing. You can add logging statements for req.body, req.params and req.query, but they're all empty. Next.js does not process our request body. So that's something we're gonna fix.

Hooking up body-parser

We'll be using body-parser to read the request body. Install the dependency, and add the following 3 lines to the top of your file:

import bodyParser from "body-parser";
import util from "util";

const getBody = util.promisify(bodyParser.urlencoded());

urlencoded() returns an middleware function that has the common 3 arguments, request, response and next. We use promisify to turn the callback function into an async function, because we won't be using this as a traditional middleware.

If you'll now update getServerSideProps and add await getBody(req, res) to just before that log statement, you'll see that our request body has been processed. 🤯

export async function getServerSideProps({ req, res }) {
  await getBody(req, res);
  console.log(req.method, req.body); // POST { name: 'smeijer' }
  // …

And now you're able to use getServerSideProps to handle your mutations. Seriously, do with the data in req.body, whatever you like. You can access secrets from process.env, and connect to the database in getServerSideProps just fine. It only runs on the server, and won't be exposed to the client.

Full Example

Here is the full, runnable, Next.js page file. You'll notice that I've wrapped the body parser statement inside a check against the request method. That's technically not required, but let's just add the body parsing in there so it doesn't do unnecessary work during GET requests. Besides, we're likely to add more logic in the POST handler, like writing stuff to our database.

Depending on your needs, you can return completely different props, or a similar shape. To keep things predictable, I definitely recommend the latter.

import bodyParser from "body-parser";
import { promisify } from "util";

const getBody = promisify(bodyParser.urlencoded());

export async function getServerSideProps({ req, res }) {
  if (req.method === "POST") {
    await getBody(req, res);
  }

  return {
    props: {
      name: req.body?.name || "smeijer",
      message: req.body ? "received!" : "",
    }
  };
}

export default function IndexPage(props) {
  return (
    <>
      <form method="post">
        <input name="name" defaultValue={props.name} />
        <button type="submit">submit</button>
      </form>
      <p>
        {props.message}
      </p>
    </>
  );
}

Final word

Even though Next.js doesn't advertise the getServerSideProps handler in this way, I think it can be very useful as a simple way to add forms to your page, without the need to define api routes, and deal with client state.

I also haven't tried if this works when hosting on vercel, as I'm running my Next.js apps on my own VPS instead of "serverless". So if anyone can give that a shot, and let us know if it worked out, that would be great.

👋 I'm Stephan, and I'm building metricmouse.com. If you wish to read more of mine, follow me on Twitter.

Typescript Type Assertions

Stephan Meijer — Tue, 07 Sep 2021 15:29:28 +0000

Type assertions look a lot like of type guards, with the exception that they don't need to be embedded in an if statement.

Imagine we have a blog, and allow authenticated users to post comments. We come up with a function like this:

function isAuthenticated(user: User | null) {
  return typeof user !== null;
}

function addComment(user: User | null, comment: string) {
  if (!isAuthenticated(user)) {
    throw new Error('unauthenticated')
  }

  db.comments.insert({ author: user._id, comment });
}

We have extracted an isAuthenticated helper that tells us if the user is logged in. And we make sure to throw an error if they are not.

In pure JavaScript, we would be done by now. An error will be thrown if user is null, so by the time we reach the database statement, we're sure that the user object is defined.

TypeScript on the other hand, still sees the user as User | null. To fix that, we can introduce a type guard. Update the helper by adding the type predicate user is User , and it understands that user is null in the scope of the if statement, and thereby has to be User after it.

function isAuthenticated(user: User | null): user is User {
  return typeof user !== null;
}

I recommend reading my earlier article about Type guards and Type predicates if you're unfamiliar with those.

By adding the type predicate we've fixed the issue in the database statement. TypeScript is aware that user will never be null at that stage.

Assertion Functions

The problem lies in repetition. Having those 3 lines of code all around the project, adds noise. Besides, it's unlikely that this is the only check you have defined.

We could turn that check into something like:

function assertAuthenticated(user: User | null): user is User {
  if (user === null || user === undefined) {
    throw new Error('unauthenticated');
  }

  return true;
}

function addComment(user: User | null, comment: string) {
  assertAuthenticated(user);

  db.comments.insert({ author: user._id, comment });
}

In plain JavaScript, that would still work. The assertAuthenticated function throws if the user object is not defined, and because the error propagates, we never reach the database statement.

However, because we removed the wrapping if statement, TypeScript is again not happy. The user._id in the database statement, will throw an TS2531: Object is possibly 'null'. To fix that, we'll insert the Type assertion.

It's quite trivial, really. Simply add asserts in front of the type predicate, and remove the return statement from the assertion function. Where type guards must return a boolean, assertion functions must return void.

function assertAuthenticated(user: User | null): asserts user is User {
  if (user === null || user === undefined) {
    throw new Error('unauthenticated');
  }
}

And now when you call this function, TypeScript knows that the value of user can never be null after that line.

function addComment(user: User | null, comment: string) {
  assertAuthenticated(user);
  db.comments.insert({ author: user._id, comment });
}

Assert Generic

Now that we know about asserts, we can also quite easily introduce a reusable helper function:

function assert<T>(
  condition: T,
  message,
): asserts condition is Exclude<T, null | undefined> {
  if (condition === null || condition === undefined) {
    throw new Error(message);
  }
}

And then whenever you have a function that accepts optional or partial values, you can simply protect them using a this assert helper.

async function latestBlog() {
  const blog = await db.blogs.findOne({ author: 'smeijer' }); // Blog | null
  assert(blog, 'author does  not have any blogs');

  // and here we know `blog: Blog`
}

When the blog is not found in the database, the assert statement will throw an error up the chain, if it did find something, we can safely work with the object after the assert call.

The next time you consider type casting a property with as MyType, consider writing an type assertion instead. Instead of simply silencing TypeScript, you'll get runtime validation with a single line of code.

👋 I'm Stephan, and I'm building metricmouse.com. If you wish to read more of mine, follow me on Twitter.

How I got a job offer from a funded startup

Stephan Meijer — Thu, 06 May 2021 14:23:23 +0000

I recently got an exciting job offer, that I had to decline. This article is my attempt to share how it came to that, and what I've learned from it.

How it started

It all started with a simple tweet about a tool that I wanted to share with all of you. As the company in question asked me not to mention them in this article, I need to keep this intro a bit vague. But, that's fine. This article is about the experience, not about "the startup" in question.

Shortly after that tweet, they send me a DM. They are trying to build a community around the tool and were wondering whether I remembered how/where I learned about its existence. After a few messages back and forth, they invited me to their Discord.

The invite

There weren't many users in the Discord, and we were just chatting a bit about random dev stuff. In the meanwhile, I was reading a bit about the startup and found out they were looking for people. I didn't consider switching jobs at the time. But as a maker, I was curious about how the development of an open source cli utility contributes to building a sustainable business.

They explained me that there hasn't been an official announcement, but that they were working on an companion product. They also mentioned that they've noticed that I've "built some impressive stuff like updrafts.app", and were happy to chat if I was interested in any of the positions.

I never expected that my question about their business would lead to an invite to chat. I know this is something small. But I had a moment of pride. Honestly, hearing or reading someone else say that they noticed your work, and found it impressive, feels amazing. I wish people would compliment each other more often. I'll never know for sure, but I like to think that it was my side projects that opened this door.

Anyways, I wasn't looking to switch jobs. So that's where the story ended.

Let's talk

Fast forward to April '21. They send out a tweet mentioning that they were looking to expand the team. As I also operate on the hiring side, it's interesting for me to know what vacancies are out there, and what financial compensation companies offer. I see reading vacancies as part of my job, but I never apply as part of it. I want to stay respectful of people's time. Going into an interview when I know there is no chance of success, is a waste of time and I find that disrespectful.

Their vacancies mentioned "Competitive salary and meaningful stock options". So I went to Discord and asked if they had more specifics about that. Almost instantly, they replied with their salary range, and the range of stock options one would receive. That message was later removed. Although I don't agree with hiding salaries, I won't quote it out of respect to them. In that same minute, they also mentioned that they were "happy to hop on Zoom".

It took me 20 minutes to come up with a proper response to that message. As I mentioned, I usually don't jump into calls, because I don't want to waste time if I already know it won't be a fruitful talk. But something was different this time. One of the vacancies was like it was written towards my profile, and their salary range looked promising. The startup felt like an awesome company to work for. I was honestly interested, so I decided to send them a message to let them know I was ready to chat.

First chat

And that was honestly all I expected. We scheduled a 30-minute chat, and I expected that we would talk about what kind of company the startup is, and what roles they were looking to fill. Time was flying by, and our 30-minute chat turned into a 3-hour talk. I got really excited about this!

We talked a bit about the startup, how they plan to grow it into a sustainable business, and we talked a bit about my day job. I think those subjects would have fitted perfectly within that 30-minute window. If only we didn't start the conversation around my open source work, side projects, and articles that I've been publishing.

During this first chat, it became clear that we were a perfect match. I'm able to bootstrap applications, I'm independent, can move fast, and have experience with early-stage startups. I check all the boxes in the profile that they're looking for. I don't mean it to brag. We were just so compatible, that I got more and more excited about this by the minute. It was like love at first sight but on a professional level.

We ended our call with the notice that we would both digest the conversation, and would come back to it later.

They bought a license!

A day after our call, I noticed that they bought a license for updrafts.app, and created an account at rake.red. I reached out to them on Discord, mentioned that I noticed their purchase and that I'm happy to jump on another call to walk them through what it has to offer.

They mentioned they'll keep digging for a while, but would like to schedule the next call to continue our conversation from the day before. And so we did.

That second call didn't last long. They mentioned that they were impressed by my work. They liked how I'm able to solve my own problems and turn them into products that can be monetized. They found my articles structured, well written, and I'm able to explain complex subjects in simple terms. I planned to ask them what they thought of the products, but it completely slipped my mind.

The call took about 15 - 20 minutes, and we ended with an agreement that I'd do a test task.

Test Task

Even though they were honestly excited about my side projects, open-source work, and articles, it wasn't enough to exempt me from doing a test task. Their task involved "Creating a prototype of an interactive API documentation that can be used for user testing". The description was a bit vague, and if I already had the job, I'd definitely jump on a call to get a better understanding of what they had in mind. But I approached it as a chance to show that I'm able to turn a vague customer description into a product. I believe that's what distinguishes me from most other developers.

I did ask them how much time they'd expect candidates to spend on the task, to which they replied that that "It’s very individual, typically 3–10 hours".

Honestly, creating something useful in 3 hours sounds impossible to me. But I was confident that I could create something in ~8 hours (a typical workday). And so I did. I tweeted about my prototype, and they send a DM that they liked what they were seeing.

Nailed It!

Exactly a week after the second call, we had our third call. We talked about the test task, and I explained how I interpreted the task and why I've built the site the way I did. They mentioned that I found the right balance between trade-offs, spend my time on the right parts, and kind of nailed it. They also mentioned that ~8 hours is the sweet spot to spend on a task like this.

When we were done talking about the task, I came back to that forgotten question from call 2. "You've tried my products, what do you think of them?" I got some useful feedback out of that, and some more compliments. Which isn't why I asked about it, but it felt nice nevertheless.

Shortly after the third call, I had a fourth to meet one of the colleagues. It was a more informal chat, meant as a "get to know each other".

Three days later they made me an offer, which I later declined due to personal reasons.

Lessons Learned

I hear you think. Okay, nice story. But why did I read this? Well, I've learned a few things, and I was working towards that. So let me share what I got out of this:

Visibility for the win! For the bigger part of my career, I've been invisible. Working as a one-man team on a business-to-business collaboration platform, while not hearing anything from anyone. Mostly triggered by covid, I've moved to platforms like Twitter and Discord to satisfy my hunger for a bit of social contact. Since then, I tweet occasionally and feel less burdened to send an innocent message like "got any details?". A year ago, I wouldn't have expected this, but seriously, it opens doors! Just make yourself visible. By showing your work, responding to people, or just a small chat. It doesn't matter how, but stay respectful. It's not about having thousands of followers either. It's about showing who you are.
I believe I nailed this one because I'm confident that I'm a match. I also know how to screw up, but overall I have a good score of hit/miss ratio. One thing that keeps coming back, is that I fail at the interviews that come up with white-board or trivia questions. But prepare a drink, and chat with me about my experiences, apps that I've built, ask me how I would solve a customer's problem, and I'm able to convince you of my strength. Not because I'm a show-off. But because I'm confident in what I do.
Test Tasks suck! It took 8 hours of my time. Trivia sucks more, but this ain't awesome either. In my case, it was okay. I liked this task, and I can see myself building it into another (open-source) side project. The whole experience was motivational, and I got tons of new ideas. But I would never ask a candidate to do the same. Especially not as vague as this one was. And there it was again. The Trivia. Should I ask for more details to build exactly what they expect? Or would the simple fact of asking, make me fail this round?
When possible, stay in touch! I think one of the nice things about this experience was that it was just so informal. I didn't wait for the next call to let them know I noticed their purchase of one of my apps. And they didn't wait for the next call to let me know that they liked my task result. This back and forth messaging between the calls, has a binding effect. It strengthens your relationships and thereby improves your chance of success. But again, don't overdo it. Stay respectful and professional.

Final Words

The startup felt like an awesome company to work for. I didn't accept the position, but I can imagine that there will be days that I'm thinking back to this experience with the thought that I've made the wrong choice.

The calls were nice, and the task was highly relevant for the position that I was applying for. If you're looking for a job, I recommend taking a look at...

Sorry, this is where it ends. They asked me to remove all references to their company and personal names. So I can't point you to their jobs page, or discord. Follow me instead? You can find me on Twitter and my other articles are listed at meijer.ws/articles.

Trivia or Job Interview?

Stephan Meijer — Tue, 04 May 2021 20:38:38 +0000

I've built multiple SaaS, and some are used by multinationals. Yet, I fail miserably at tricky interview questions. In this article, I'm going to show you a few recent questions I got, and share my thoughts.

In case you haven't seen any of my work. In the last 12 months, I've launched testing-playground.com, updrafts.app, rake.red and a bunch of open source projects. I like to believe that I know what I'm talking about.

Question 1

Here is a simple design of a navbar menu, we are struggling to stick the login button to the right edge of the <nav> (borders are added for a better understanding).

<nav>
  <a href="#">Home</a>
  <a href="#">Products</a>
  <a href="#">About</a>
  <a href="#">Log in</a>
</nav>

<style>
  nav { display: flex }
  a { margin: 0.4rem; padding: 0.4rem; }
</style>

Options - Single Choice

a:last-of-type { margin-left: auto; }
a:last-of-type { margin-left: 100%; }
a:last-of-type { float: right; }
a:last-child { margin-left: auto; }
nav:last-child { float: right; }

My thoughts

The interview existed of 12 questions, that had to be answered within 16 minutes. That sounds doable, until you meet the trick questions.

Have you thought about the question? I think that a:last-child { margin-left: auto; } is the correct answer. But a:last-of-type { margin-left: auto; } works as well. As this is an automated interview, I can only hope that I choose whatever the interviewer prefers. As a:last-of-type is listed first, this is easy to get tricked by. Time is limited, so when the candidate is confident that an option works, they are going to choose that, and move on to the next question.

I usually tend to group the left and right options, and add a justify-content: space-between. Why is that not an option? Is that wrong? Can we talk about it?

Question 2

const raiseError = (message: string) => {
  const err = new MyError(message);
  throw err;
};

What is the return type of this function in TypeScript?

Options - Single Choice

MyError
void
null
undefined
never

My thoughts

The clock was ticking, and I chose void. As that's how I would annotate this function. Makes sense, right? Think again! The return type of this function isn't void, it's never. As it's impossible for this function to return.

To make it return void, the throwing should be conditional. (wrapped with an if)

I believe this question is wrong because it doesn't say much about your TypeScript experience. I mean, how many dedicated throw functions do you have in your codebase? Most functions are constructed in a way that they have a return path. Either with a value, undefined, or void.

And when you do come across this edge case in your day job, how hard would it be to place your cursor at the function, and wait for that pretty tool-tip to appear, telling you the exact return type?

Question 3

Which HTTP methods are idempotent?

Options - Multiple Choice

All of them are idempotent as it is a stateless protocol
None of the HTTP methods are idempotent.
All of them except for POST, CONNECT and sometimes PATCH.
All of them except for POST, OPTIONS and TRACE.

My thoughts

First of all, this was a test for a frontend developer position. Are frontend developers really expected to know if certain HTTP methods are idempotent or not? Isn't that for the API developers to know? I really did not know the answer to this question.

After the test, I found out that the HTTP spec does have this specified, while I assumed that it was for the API spec (like open-api) to decide.

Anyways, I guessed that all of them are idempotent, as HTTP doesn't hold state. My database does. But according to MDN, it should have been all of them except for POST, OPTIONS, and TRACE. Today I learned.

Now the question is, what if I make my POST handler idempotent? Doesn't this question depend a tiny bit on the API that we're talking about?

Question 4

Which of the following browser actions/events are triggered by changing the CSS property opacity?

Options - Multiple Choice

Layout operations performed
Painting/Rasterizing
Page composited together
None of the above

My thoughts

Seriously? For what would we need this? I guess that the browser does a repaint, so that excludes the last option. Opacity doesn't change the layout, so there wouldn't be any layout operations. But what about Page composited together? I don't know. I really don't. Does that make me a bad developer?

Let's move on. Do you still want that opacity on your navbar?

Trick Questions

There were six more weird technical questions that made more or less sense than the four above. But they all had one thing in common. It felt like they were trying to trick me, and it are questions that I don't need to know to be able to develop solid applications.

When I would need to know it, I'm able to open my browser, and find the right answer in a matter of minutes. I'm a developer, but I suck at trivia.

Two more questions to wrap this up? Remember... your time is ticking:

Question 11

How would you explain a complicated technical problem to a colleague having none to very little technical understanding?

Write answer here...

Question 12

How would you go about getting a buy-in for your project from multiple stakeholders at work?

Write answer here...

I don't know what you want me to say. Do you have more details? Can I get another coffee and 30 minutes of your time? Let's talk about it.

My score

The "nice" thing about this automated test, is that you'll get your score right away. I had a total score of 47%. I suck at React, HTTP, Communication, well, basically in every area.

As expected, a few hours after my submission, I got the following mail. A little surprised that that part wasn't automated as well.

Hi Stephan,

Thanks for doing the skills test! It was a tough decision as we've had so many great applications, but we've decided to move forward with other candidates. Really looking for folks who've got more experience with javascript.

Well, this really was a motivational experience (not). Do you know the saying "You've dodged a bullet?". That's how I came to think about this kind of interview tests.

As I also operate on the hiring side, this provides me valuable insights into how the industry works. But if any recruiting person is reading this, please stop it. You're hiring developers specialized in interviews. Not in creating awesome software.

A quick dive into generators

Stephan Meijer — Thu, 15 Apr 2021 07:36:25 +0000

I've briefly mentioned generators earlier in my article about recursion. Today, I'm going to explain the concept of generators to you, and why I believe that they are an important thing to know. If you haven't read that article, I'd recommend doing so, as this explanation builds upon that one.

Introduction

Let's take the recursive function and the recursive generator function from the earlier article. Both these functions convert a tree-like structure to a flat list where each item has an id and a parent property:

The recursive function looked like:

function flatten(node: Node, parent?: Node): FlatNode[] {
  const nodes: FlatNode[] = [{ id: node.id, parent: parent?.id }];

  if (Array.isArray(node.children)) {
    for (const child of node.children) {
      nodes.push(...flatten(child, node));
    }
  } else if (typeof node.children === 'object') {
    nodes.push(...flatten(node.children, node));
  }

  return nodes;
}

While it's generator variant looked like:

function* flatten(node: Node, parent: Node): Generator<FlatNode> {
  yield { id: node.id, parent: parent?.id };

  if (Array.isArray(node.children)) {
    for (const child of node.children) {
      yield* flatten(child, node);
    }
  } else if (typeof node.children === 'object') {
    yield* flatten(node.children, node);
  }
}

Now, most of my projects have an utility that I named ensureArray. It's a nifty little helper that wraps values in an array, unless it already is an array. Something like:

function ensureArray(object) {
  if (typeof object === 'undefined') {
    return [];
  }

  if (Array.isArray(object)) {
    return object;
  }

  return [object];
}

I share this because this little utility lets me clean up these functions and make the similarities more obvious. I'll also stop annotating the examples with types, to further reduce the noise.

Recursive generators

In case you've never seen generators before, (overly simplified), generators are functions decorated with an * and using the yield keyword to return values. There is a lot to read about them, but the nice thing is that they are executed lazily. Meaning, when we call flatten here, it's possible to only process the first n nodes, and ignore the rest. Where the non-generator variant would first process the entire tree, only to discard everything afterward, generators allow us to only process the absolute minimum of what's required for the task at hand.

We'll come back to that. Let's take a look at the implementation first. I've simplified the examples from above using the ensureArray helper, and I've added a log statement:

Recursive function:

function flatten(node, parent) {
  console.log('flatten', node.id);  
  const nodes = [{ id: node.id, parent: parent?.id }];

  for (const child of ensureArray(node.children)) {
    nodes.push(...flatten(child, node));
  }

  return nodes;
}

Recursive generator:

function* flatten(node, parent) {
  console.log('flatten', node.id);
  yield { id: node.id, parent: parent?.id };

  for (const child of ensureArray(node.children)) {
    yield* flatten(child, node);
  }
}

You see the similarities, right? I hope that makes it less daunting.

Instead of adding the node to an array, we directly yield (return) it, and instead of pushing nested nodes to that same array, we also yield those. The * that you'll see behind that second yield, is syntactic sugar to yield all results in an array/iterator individually.

yield* flatten(child, node);

could just as well be written as:

for (const result of flatten(child, node)) {
  yield result;
}

Lazy evaluation

So the thing I mentioned earlier about the lazy behavior? Imagine we need to do something only for the first three nodes in that tree. We would write something like this:

const nodes = flatten(tree);
for (let idx = 0; idx < 3; idx++) {
  console.log('handle', nodes[idx].id);
}

Using the traditional, non-generator approach, this would result in the following log:

flatten 1
flatten 2
flatten 3
flatten 4
flatten 5
flatten 6
flatten 7
flatten 8
flatten 9
flatten 10
flatten 11
handle 1
handle 2
handle 3

That log tells us that the entire tree is processed and converted to the flat array before we can handle the 3 nodes that we need. The processing time that we used for those other 8 nodes, is wasted.

Now, if we'd do the same with that generator function, we'd need to change the syntax a bit:

const nodes = flatten(tree);
for (let idx = 0; idx < 3; idx++) {
  console.log('handle', nodes.next().value.id);
}

We no longer use the idx property, but instead, call the next function from the nodes.

The flatten call itself doesn't do much there. It does not invoke the flatten function. The log on that first line? It's not printed. Instead, the call prepares the generator and returns an object with a next method. When we call the next method, the generator will run till the next yield inside that function. When it meets that yield, it will return the value that's being yielded.

The return value of next is not just that yielded value. It's an object with a value prop, holding your yielded value, and a done property, holding a boolean that will tell you if this generator is done generating values.

So the output from that last loop?

flatten 1
handle 1
flatten 2
handle 2
flatten 3
handle 3

It's important to understand that the output order has changed. We can handle the node, as soon as the generator yields one. It doesn't yield all nodes at once, it yields every node individually, as soon as it has it. We don't need to wait for the entire tree to be processed. In fact, the processing won't continue, until we explicitly ask for the next node.

Once we've handled our three nodes, we stop our loop, and the tree is not further processed. We haven't wasted any processing time using the generator approach.

You probably don't always need loops, and sometimes you do want to process all or nothing. In those cases, it's trivial to wrap the call in Array.from, to get all nodes at once. Just like you would have with the non-generator approach:

const nodes = Array.from(flatten(tree)); // [{ id:  … }]

We've used a simple loop in this example, but you can imagine that this is quite powerful. Without changes to the generator itself, it can be wrapped with logic to only handle the first n results, or only process until a certain condition is met.

Also, isn't it just beautiful, how easy it is to write recursive functions this way? No intermediate arrays. No return complexity. Recursive tree parsing, in 3 lines. All it asks is to get familiar with yield.

function* flatten(node, parent) {
  yield { id: node.id, parent: parent?.id };

  for (const child of ensureArray(node.children))
    yield* flatten(child, node);
}

Final word

Generators might look a bit scary at first, but they come with a lot of flexibility and power. I can imagine that they look daunting, especially for inexperienced developers. But I would really recommend getting familiar with them. They make a great asset to your utility belt.

If you have questions related to this subject, please let me know in the comments. I'm happy to explain things in more detail.

👋 I'm Stephan, and I'm building rake.red. If you wish to read more of mine, follow me on Twitter or check my work at meijer.ws.

Three ways to handle recursion

Stephan Meijer — Mon, 12 Apr 2021 12:58:24 +0000

This is a follow-up post on You might not need recursion. In this article, I'm going to show you three different ways to convert a Tree data structure, to a flat list while maintaining a reference to the parent.

Let's start with the input that we're working with:

type Node = {
  id: number;
  children?: Node | Node[];
};

const tree: Node = {
  id: 1,
  children: [
    {
      id: 2,
      children: [{ id: 3 }, { id: 4 }],
    },
    {
      id: 5,
      children: [{ id: 6 }, { id: 7, children: [{ id: 8 }] }],
    },
    {
      id: 9,
      children: { id: 10, children: [{ id: 11 }] },
    },
  ],
};

As you can see, this tree has a hierarchical structure. Every node has an id, and an optional property called children which is either an array or an object.

We're going to convert this to a flat array holding items with an id and a parent property:

type FlatNode = {
  id: number;
  parent?: number;
};

const output: FlatNode[] = [
  { id: 1 },
  { id: 2, parent: 1 },
  { id: 3, parent: 2 },
  { id: 4, parent: 2 },
  …
]

Recursive function

When working with Tree-like structures like the one above, we tend to write recursive functions by default. Despite the fact that recursion is hard to grasp for a lot of us. Even amongst senior developers, with many years of experience.

When we write a recursive function to handle this, we end up with something like the following:

function flatten(node: Node, parent?: Node): FlatNode[] {
  const nodes: FlatNode[] = [{ id: node.id, parent: parent?.id }];

  if (Array.isArray(node.children)) {
    for (const child of node.children) {
      nodes.push(...flatten(child, node));
    }
  } else if (typeof node.children === 'object') {
    nodes.push(...flatten(node.children, node));
  }

  return nodes;
}

When calling flatten(tree), it starts processing at the root node and recursively walks down the tree walking over the children, to return them as a FlatNode. To be able to keep the reference to the parent, we need to pass in the parent as an additional function argument.

There is nothing wrong with this function. And I believe that it's perfectly understandable. However, my experience also tells me that I will have coworkers working on the same code base, that find this concept hard to understand.

If you haven't worked with recursion before, and think you'll understand what's going on, I want to challenge you. Take the tree object from above, and write this flatten function without looking back to my example before you have a working result.

Flat iteration

This recursive function can also be rewritten to a flat loop. The following example has the same input and output as the recursive function, but all operations take place in a single call frame. There is no recursion and there are no calls to an external function.

function flatten(rootNode: Node): FlatNode[] {
  const nodes: FlatNode[] = [];
  const queue = [rootNode];

  while (queue.length > 0) {
    const node = queue.shift();

    if (Array.isArray(node.children)) {
      for (const child of node.children) {
        queue.push({ ...child, parent: node });
      }
    } else if (typeof node.children === 'object') {
      queue.push({ ...node.children, parent: node });
    }

    nodes.push({ id: node.id, parent: node.parent?.id });
  }

  return nodes;
}

Now, I do believe that this is easier to follow for people unfamiliar with recursion. But I also think that the difference in complexity is fading. This is a more complex function than the one from my earlier article because the subject is more advanced as well.

From the performance point of view, in Chrome the recursive function is twice as fast, while in Firefox the non-recursive function is the faster one.

Also, mind that while the output has the same structure, the resulting nodes are in a different order. The recursive function eagerly moves to the child nodes and handles children before siblings. While the loop handles siblings before children. Making both functions merge their results in a different order.

Recursive generators

Generators are particularly well suited to tackle recursive problems.

In case you've never seen generators before, (overly simplified), generators are functions decorated with an * and using the yield keyword to return values.

Let's take a look at the implementation:

function* flatten(node: Node, parent: Node): Generator<FlatNode> {
  yield { id: node.id, parent: parent?.id };

  if (Array.isArray(node.children)) {
    for (const child of node.children) {
      yield* flatten(child, node);
    }
  } else if (typeof node.children === 'object') {
    yield* flatten(node.children, node);
  }
}

Now, this solution will return the values in the same order as the recursive function. In fact, they do look quite similar, except that we don't need that temporary nodes array to merge the results.

Instead of adding the node to an array, we directly yield (return) it, and instead of pushing nested nodes to the same array, we also yield those.

Final word

Whatever you prefer is fine. I think it's most important to choose the method that's most familiar to your team and most fitting to your requirements. Remember that for inexperienced developers the loop is easier to understand and that it's always the easiest one to debug.

I personally would recommend getting familiar with generators. They look a bit scary at first, but they come with a lot of flexibility and power.

👋 I'm Stephan, and I'm building rake.red. If you wish to read more of mine, follow me on Twitter or check my work at meijer.ws.