DEV Community: Jack

Dependency sandboxing in node.js with Jpex

Jack — Tue, 06 Jul 2021 21:56:20 +0000

React recap

Okay so I've written about jpex a few times, particularly in relation to react

Essentially it allows you to do something like this:

import { useResolve } from 'react-jpex';

const useMyDep = () => {
  const dep = useResolve<IDep>();
  return dep.someValue;
};

and this:

import { encase } from 'react-jpex';

const useMyDep = encase((dep: IDep) => () => {
  return dep.someValue;
})

depending on your preferred flavour.

Jpex uses the service locator pattern to resolve and inject dependencies, plus it's super-powered by Typescript inference for a super slick experience. But the really really cool thing about it is you can test your react components with the provider component to stub your dependencies:

<Provider
  onMount={(jpex) => {
    jpex.constant<IDep>(mockValue);
    // everything rendered inside this provider will use the mock value
    // everything outside of the provider will use the "real" value
  }}
>
  <ComponentUnderTest>
</Provider>

Using jpex with node.js

However, we're talking about node.js right now, not react. How does jpex work with node? Well at first glance it's pretty similar to the front end:

import jpex from 'jpex';

const getMyDep = () => {
  const dep = jpex.resolve<IDep>();
  return dep.someValue;
};

import jpex from 'jpex';

const getMyDep = jpex.encase((dep: IDep) => () => {
  return dep.someValue;
});

Easy right? The problem is that it's then quite hard to create a "sandboxed" environment. How do you call these functions with mocked values?

Option 1: mocking at the test level

it('returns some value', () => {
  jpex.constant<IDep>(mockValue);

  const result = getMyDep();

  expect(result).toBe(mockValue.someValue);
});

This method can be problematic because you're registering a test mock on the global instance. It will then be used as the resolved value for IDep everywhere in the file, unless you register it again in the next test. This sort of leaky test is a bad idea and will almost definitely cause bugs.

Option 2: only using encase

it('returns some value', () => {
  const result = getMyDep.encased(mockValue)();

  expect(result).toBe(mockValue.someValue);
});

encase actually exposes the factory function so you can manually pass in your dependencies, which means you can test it safely like this. This works well for some cases. But what if your function is called by another function?

const someOtherFn = () => {
  return getMyDep();
}

Now you cannot test someOtherFn without getMyDep attempting to resolve its dependencies!

Option 3: the composite pattern

Another pattern for dependency injection is the composite pattern. Essentially your entire application is made up of factory functions that must compose at app start. In this case you'd be passing the jpex object through your composite chain like this:

export default (jpex) => {
  return {
    getMyDep: jpex.encase((dep: IDep) => () => dep.someValue),
  };
};

I'm not keen on this myself, it kinda defeats the point of a service locator!

So if you can't actually invert the control of your dependencies, is jpex just useless in node applications? Yes... until now!

A more robust solution to DI and testing

I have just published a new library: @jpex-js/node. You use it like this:

import { resolve } from '@jpex-js/node';

const getMyDep = () => {
  const dep = resolve<IDep>();
  return dep.someValue;
};

import { encase } from '@jpex-js/node';

const getMyDep = encase((dep: IDep) => () => {
  return dep.someValue;
});

Looks familiar right? It's essentially the same syntax as jpex and react-jpex so far, and works exactly the same. The magic starts to happen when you want to sandbox and stub your dependencies...

The library exports a provide function. What this does is creates a new instance and then every resolve and encase call within is contextualised to this new instance. You can think of it as an equivalent to the <Provider> component in react-jpex.

If we attempt to write the same test as earier it might look like this:

import { provide } from '@jpex-js/node';

it('returns some value', () => {
  const result = provide((jpex) => {
    jpex.constant<IDep>(mockValue);

    return getMyDep();
  });

  expect(result).toBe(mockValue.someValue);
});

Regardless of whether this function used resolve or encase, we're able to control the dependencies it receives!

One more thing

If the idea of a sandboxed DI context in which to run your tests seems cool, I should also point out that this supports asynchronous call stacks as well. Any promsies, callbacks, or timeouts, are kept within the same context:

provide(async (jpex) => {
  jpex.constant<IDep>(mockValue);

  await waitFor(200);

  setTimeout(() => {
    getMyDep(); // still retains the context
    done();
  }, 1000);
});

Conclusion

As the author of jpex I am definitely biased but I am a big proponant of making dependency injection a core part of javascript development, but also a slick developer experience. I've been using jpex in react applications for a few years now and I love it. And now with this library, we should be able to bring the same patterns and ease of testing to node applications as well.

Behind the scenes we're using node's new async_hooks module and the AsyncLocalStorage class. It's a really powerful concept for creating and managing contexts and there is so much potential for awesome uses of it!

Why I'm not using Next.js

Jack — Mon, 05 Jul 2021 15:27:09 +0000

So I'm building an application. Not only am I using react, but I'm also using node. Surely I'm using next.js right? I mean, every other article I see seems to be a next.js tutorial, it's so in right now. Well no, I'm not using it for my project.

n.b.

This isn't just a next.js diss. A lot of people swear by it after all! I just wanted to write down my personal thought process for not adopting it.

I really struggled to find a decent pros and cons list for using it, just pros. There are definitely downsides, frustrations, and compromises if you pick next.js. I just don't feel like anybody talks about them.

On to business:

Super opinionated / abstracted

There's nothing wrong with opinionated stuff per se. But the problem is in the javascript ecosystem we don't really have any standards or best practices for anything. There are 100 ways to do everything. Want classes and OOP? Sure! Want a strictly FP app? Go for it! Want to separate your concerns and heavily modularise your application? Why not! Want to just mash everything into a monolithic gross swiss army knife component? I mean... I suppose...

This isn't like making an opinionated framework in an already-opinionated language like, say C#. If you try to make an opinionated framework in javascript, you're not saying "this follows and enforces the best practices of the language", you're saying "my way or the highway!"

This first point is basically relevant to every other point in this piece so I won't spend any longer on it.

Webpack

Oh man I'm only just starting to move away from webpack and now this framework is forcing me to use it! Not only is it forcing me to use it, but it's doing its best to stop me from configuring it too.

Weighty pages

Next effectively forces you to do all of the "stuff" at the top level pages. Anything "next-y" or "server-y" has to be done directly in your pages. This creates all kinds of coupling. Not to mention your pages are also tightly coupled to your routing. This means your routing logic is tied to your page which is tied to all of your server side, data fetching, caching, everything, all tightly coupled.

I've always believed that routes should be lightweight and only concerned with which component is rendered by which route. But now I have to also stuff all of my business logic and fetching activity? No thanks.

Server routes are hardocded to /api

Next.js forces you to mount all of your backend endpoints behind /api/. This goes back to my very first point about this being a super duper opinionated framework, where you can't even control your endpoints. I really don't want my UI framework to get to say where my api endpoints should live in my application...

Every fetch/mutation must be done in a "next-ey" manner

Everything has to be written in a way that accounts for 2 completely different rendering methods, so it can run on the server, but also on the client.

Once again, I can't just write my code in a way that suits my architecture or my conceived best practices.

Exposing low level operations

"You can now directly query the database in your react app, that's so cool!" - is that cool or is it scary? This is just asking for lazy people to directly import their db connection in a component and query it. I can't imagine some of the terrible things people have done with this kind of freedom...

I forgot I needed this bit of data. I really can't be bothered to create an abraction that retrieves the data I need in a sensible, testable context...

import { db } from '@/db';

const bitOfData = db.query(...)

I'll totally come back and refactor this later.

So now we're abstracting bits that, to me, don't need abstracting, like routing; and exposing low level backend functionality that we should definitely not be tying into the ui? Oh man.

Gotchas

There are gotchas everywhere:

For the initial page load, getInitialProps will run on the server only. getInitialProps will then run on the client when navigating to a different route via the next/link component or by using next/router. However, if getInitialProps is used in a custom _app.js, and the page being navigated to implements getServerSideProps, then getInitialProps will run on the server.

This is just one piece of confusing documentation that has tripped up friends of mine. I hear stories from them every day about weird edge cases and gotchas and "because next says so" issues. Every day I feel more validated that I don't have to deal with these frustrations.

BUT SEO!

SEO is not stupid - even without Next. With a page that just uses client-side rendering and has to fetch some data first, Google's page crawler is not stupid, it knows to wait for this sort of thing. As long as I'm adding the necessary meta tags I can more of less trust that SEO will still work.

The only major cost of not using next.js is that I can't send contextual link previews on facebook/twitter/slack/etc. They don't process javascript so I can't dynamically add meta tags to the document.

However, I don't think rewriting my entire application from the ground up, in a framework I find constrictive, just to get this single feature, is worth it. There are many many ways around this, like pre-rendering for robot useragents.

Conclusion

So there you go. The tl;dr is that I'm a stubborn developer and next.js isn't the boss of me!

So if you're not using next.js I guess you'll go back to CRA then?

Nope! I've only ever used CRA once and I ejected out of it after a couple of weeks. I quickly found that I couldn't do some of the more advanced stuff I wanted to do. Decisions were taken out of my hands that I didn't want to give up. Like typescript was being transpiled by a webpack loader rather than babel, but I use several babel plugins that parse typescript annotations. I also quickly realised there was - so - much - stuff - in the webpack config that I was just never ever going to need. I stripped out hundreds of lines of code and unneeded dependencies.

For this current project I decided to use vite. It's still somewhat opinionated and it's super-low config but I've not yet hit an issue I couldn't easily solve with a little config.

Sharing contracts

Jack — Thu, 03 Jun 2021 06:32:18 +0000

This is a concept that really drives home how nice it is when your entire stack is written in the same language. I am using node/typescript on the backend which means I can share stuff between the ui and api.

I have a separate repo that holds common stuff like error codes, request payloads and response payloads:

// Contracts
export interface SearchProductsRequest {
  page: number;
  q: string;
}

export interface SearchProductsResponse {
  nextPage: number;
  products: Product[];
}

Then I can use that to bridge the gap between api and ui:

// API
async function searchProductsApi(
  req: Request<void, void, void, SearchProductsRequest>,
  res: Response<SearchProductsResponse>
) {
  // req.query is strongly typed
  const result = await search(req.query);

  // res.send will throw a ts error if result doesn't match the response
  res.status(200).send(result);
}

// UI
// (driver is a simple wrapper around the fetch api)
async function searchProducts(args) {
  // data will be strongly typed
  const { data } = await driver<SearchProductsRequest, SearchProductsResponse>({
    url: "/products",
    // will throw a ts error is args doesn't match the request
    data: args,
  });
}

Now both sides of my application can have confidence that they will be receiving/sending the same payloads!

Developing a ui architecture

Jack — Fri, 28 May 2021 11:04:18 +0000

So previously I discussed at length how my backend application is archtected. If you haven't read it already I'd strongly suggest you do before continuing, as this article very much continues on the same path.

So once again my frontend is based on hexaganoal architecture. It's very very similar to the pattern I explained previously, with a few small but important differences.

When we talk about frontend we always focus on component organisation, and this is important, of course; but it also just as important to think about organising the non-presentational aspects of your application. I often see people making fetch requests inside components. This might be quicker to throw together and push out, but as a project grows, it becomes an absolute chore to find where in your code you are fetching data or causing side effects.

Overall Structure:

src
│
└───ui
|
└───application
|   └───feature
|       |   useUsecaseHook.ts
|
└───core
|   |   feature.ts
|
└───infrastructure
|   └───feature
|       |   method.ts
|
└───domain
|   |   feature.ts

The most obvious difference is the api layer has been replaced with a ui layer, for obvious reasons. I'll come back to the contents of the ui layer shortly.

Application

The other difference is that the application layer is actually now just a collection of react hooks. For some reason in a react application, this just makes a lot of sense. All of your usecases are going to be tied to hooks and state. The only problem with this approach is the application layer is coupled to react so you couldn't access any of the usecases outside of a react context. However, I decided this was a small architectural price to pay for convenience (given that I'll almost definitely never be using the application layer outside of react).

Core / Infrastructure / Domain

I won't go into these in much detail because they are literally the same as the same areas in the backend app.

The tl;dr: core is abstract interfaces, infrastructure is implementations of those interfaces (stuff that does side effects and "contacts the outside world"), and domain is pure business logic.

UI

So what's going on in the ui layer? It's nothing extraordinary, in fact it follows a pretty common pattern that's similar to atomic design:

ui
└───elements
|   |   ButtonComponent
|
└───modules
|   └───feature
|       |   CompositeComponent
|
└───pages
|   └───feature
|       | FeaturePageComponent
|
└───app
    | AppComponent

elements

Elements are small self-contained components that have no application logic or knowledge. Things like buttons, grids, inputs, and so on.

I have maybe 2 exceptions to the rule here, which is an Image component that takes a partial src and computes the full url based on my app config. And an Upload input element that internally handles uploading a file to the server and just returns the resulting url. Should I make these dumber and less tied to the rest of the stack? Yes. Will I? Maybe eventually 👀

modules

A module is a group of elements that make up part of a page. For example, if you have a search page, you might have a module for the search input area, a module for the list area, and a module for the individual list item. A module can also be made up of other modules.

A module can have domain knowledge.

The important part to note here is that modules are all "dumb". A module will never fetch data or send data, it won't read cookies, it won't use the application layer. Anything "smart" is done by the parent pages.

What complicates this is that sometimes a module might render another module that relies on some smart stuff:

function Overview({ item, onAddToBasket, onViewMoreInfo }) {
  return (
    <ProductItem
      item={item}
      onAddToBasket={onAddToBasket}
      onViewMoreInfo={onViewMoreInfo}
    />
  );
}

function ProductItem({ item, onAddToBasket, onViewMoreInfo }) {
  return (
    <div>
      <span>{item.name}</span>
      <ProductActions
        item={item}
        onAddToBasket={onAddToBasket}
        onViewMoreInfo={onViewMoreInfo}
      />
    </div>
  );
}

function ProductActions({ item, onAddToBasket, onViewMoreInfo }) {
  return (
    <div>
      <Button onClick={onAddToBasket}>Add to basket</Button>
      <Button onClick={onViewMoreInfo}>More info</Button>
    </div>
  );
}

If we want to provide the onAddToBasket prop to the ProductActions component, we have to pass the prop all the way down. Prop drilling is a frustrating and tedious part of react development. This is why we often just bypass the smart/dumb construct and stick the smart logic inside ProductActions instead, but this causes more issues as you start to lose track of where your smart behaviour comes from.

My solution is to actually pass elements as props, so you compose your dumb components and then pass them down instead:

function Overview({ item, children }) {
  return (
    <ProductItem item={item}>
      {children}
    </ProductItem>
  );
}

function ProductItem({ item, children }) {
  return (
    <div>
      <span>{item.name}</span>
      {children}
    </div>
  );
}

function ProductActions({ item, onAddToBasket, onViewMoreInfo }) {
  return (
    <div>
      <Button onClick={onAddToBasket}>Add to basket</Button>
      <Button onClick={onViewMoreInfo}>More info</Button>
    </div>
  );
}

And at the smart level (i.e. pages) then you can do:

<Overview item={item}>
  <ProductActions
    item={item}
    onAddToBasket={handleAddToBasket}
    onViewMoreInfo={handleViewMoreInfo}
  />
</Overview>

This does get more complex when you have multiple components to compose, but I think it's better than mountains of prop drilling or smart components buried deep in the module layer.

pages

A page is what it says on the tin, it is the construct of an entire page or view. Pages serve three purposes: they put together multiple modules into a cohesive whole; they handle interacting with the application layer for fetching and mutating data; and they orchestrate the routing of the application.

// A single page, composing the view from multiple modules
function ProductPage() {
  const item = useFetchItem();
  const addToBasket = useAddToBasket();
  const viewMore = useViewMore();

  return (
    <Overview item={item}>
      <ProductActions
        item={item}
        onAddToBasket={addToBasket}
        onViewMoreInfo={viewMore}
      />
    </Overview>
  );
}

// A Page that stitches together other pages with routing
function ProductPages() {
  return (
    <Route path="/product/:id">
      <ProductPage/>
    </Route>
  );
}

So there it is. Just like the backend - hexagonal architecture, separation of concerns, and dependency injection form the basis of the codebase. Is it perfect? No. Is it easy to maintain and follow? I think so. Is it for everyone? Probably not!

Developing an api architecture

Jack — Fri, 21 May 2021 21:46:43 +0000

Introduction

I love a bit of architecture. How you structure your application is so important and if you get it wrong you'll really pay for it down the line. The problem is you often don't know you've got it wrong until it's too late. I've made this mistake so many times and iterated on solutions so much that now when I start a project I know exactly how I want it to be structured. I've developed what I consider to be a pretty robust pattern for application development.

Essentially I follow a loose Hexagonal Architecture pattern. I won't go into too much detail on what hexagonal architecture is as there are tonnes of articles on the concepts and ideas already. What I will do is show how I interpret it into my own applications. Although there are lots of articles about this pattern, they are very rarely discussed in terms of a node or front end application (usually they are Java-based).

For this article I'll focus on my node app. I'll cover the front end separately as although it's similar at the core, there are some necessary differences.

This is roughly how my codebase is structured:

src
│
└───api
|   └───route
|       |   get.ts
|       |   post.ts
|       |   delete.ts
|
└───application
|   └───feature
|       |   usecase.ts
|
└───core
|   |   feature.ts
|
└───infrastructure
|   └───feature
|       |   method.ts
|
└───domain
|   |   feature.ts
|
└───bootstrap
    |   setup.ts

This is also termed Ports and Adapters:

application = usecases
core = ports
infrastructure = adapters

So what do all of these layers mean?

I've drawn a diagram of how this application fits into hexagonal architecture. Unfortunately diagram creating is not my strong point so I apologise in advance:

This looks like every other HA diagram I've ever seen and until you "get it" I don't think it really helps that much. I often find it easier to convey the flow of data like this:

At this point you might be thinking: "why are there so many steps for this one request?" and you're right. In a "basic" express app your flow would probably look more like this:

But the problem with this is you're tightly coupling your application in every sense. You make it hard to test the individual parts, you tie your application logic into your server, you tie your database into your application logic, which in turn ties your database into your server. One of the fundamental principles of good software design is to separate your concerns.

So yes this is more files and more layers of abstraction, but that's a good thing I promise!

Let's dive a little deeper into each of these folders:

api

My api layer contains my express routes and nothing else. You can think of this layer as being controllers in an MVC framework. The routes don't contain any logic, they purely pass the request data on to the application layer and then respond with the result. Not only does this keep the routes lean, it keeps all of my application logic agnostic of the delivery method.

async function(req: Request, res: Response) {
  const basket = await getBasketUsecase(req.userId);
  res.status(200).send(basket);
}

The structure of this folder mirrors the express paths, for example: /src/api/basket/get.ts equates to a GET request to /api/basket. When setting up the application, I automatically find all of the files in this folder and calculate the express routes dynamically. This means I never have to manually write app.get('/api/basket') as it's just inferred. This kind of auto-routing is quite common with big frameworks and things like next.js. Personally I like it and it feels like "magic" without being too "behind the scenes".

application

These are my use cases. What I mean by this is that each method is an end-to-end feature. For example "get the basket", "add something to the basket", "remove something from the basket". Each use case will handle things like validating inputs, calling the necessary methods to carry out the action, validating the response, transforming data into the output type, and so on. Essentially this is the "orchestration" layer of the application.

async function usecase(args) {
  await validateArgs(args);
  const data = await fetchData(args);
  const output = normalizeData(data);
  await validateOutput(output);
  return output;
}

There is almost always a 1:1 relationship between the api layer and the application layer. An api endpoint will only call one usecase, and a usecase will most likely only be used by one api endpoint. Why not just combine them into a single function? Loose coupling.

For example, although I am using express for my server, I may want certain use cases to be accessed via a CLI instead/as well. The application layer does not care if a request comes via the web api, or the cli, or some other method. It just cares about the arguments it receives.

The application, core, and infrastructure layers are hard to talk about in isolation (which is ironic) so the next few sections will be a bit intertwined...

core

How does the application layer actually "do stuff" though? If we want to get the basket, for example, how does it do this? We wouldn't want the application layer to import the database and query it directly, this would couple our low level implementation too tightly to the high level use case.

The core layer holds interfaces for all of the things the application can do. When I say interfaces, I mean typescript interfaces, there is no actual javascript here, purely types and interfaces.

So for example, if we want to get the basket, there will be a FetchBasket type that might look something like this:

export type FetchBasket = (userId: string) => Promise<IBasket>;

Our application layer operates purely on these interfaces, at no point do we import a fetchBasket function. Instead, we import the interface from the core layer and use dependency injection to say "please fetch the implementation of this type". Dependency injection is really the glue that connects these layers together.

For example, our get basket use case might look something like this:

async function getBasketUsecase({ userId }) {
  const fetchBasket = jpex.resolve<FetchBasket>();
  const basket = await fetchBasket(userId);
  return basket;
}

This means there is a "bridge" between the application layer and the underlying implementation detail, which is really important. The above function is really easy to test because the fetchBasket implementation does not exist, you can provide any implementation you want. It also means that your usecases are really clean, because all of the leg work is abstracted away and all you have to do is say "I'd like the implementation of this type please" and off you go.

One awesome benefit of this is that you can write your core layer, then your usecase layer, and not even bother with the infrastructure layer until later. This is fantastic for working on new features where you know what the use case is ("the user wants to see their basket"), and you know roughly what the interface will look like ("pass the user id to the database, get the basket back"), but you're not quite sure on the implementation detail yet.

infrastructure

Now that we have our core interfaces, the infrastructure layer contains all of the implementations for them. Essentially anything that causes a side effect, or reaches outside of your code (like accessing the database) is infrastructure.

Interestingly, infrastructure methods can rely on other core interfaces which means you can have several levels of abstraction. For example, the fetchBasket implementation will probably depend on an IDatabase interface, which in turn would be a wrapper around your actual database.

As mentioned earlier, I use dependency injection (specifically the service locator pattern) to register these infrastructure methods:

jpex.factory<FetchBasket>((db: IDatabase) => (userId: string) => {
  return db.collection("basket").find({ userId });
});

bootstrap

The bootstrap folder isn't even a layer, and it does what you think it does. We call a setup function at app start. This creates the express server, finds and registers all of our api routes, finds and registers all of our infrastructure methods, connects the database, and so on.

Misc

There are a couple of notes I wanted to add/clarify as well:

I should mention that I follow a loose function programming paradigm. You won't see any service/repository classes or anything like that. Everything is a function that depends on other functions. I've found that repository classes often become unwieldy, hard to maintain, cluttered with dependencies, and difficult to mock. (Plus all data is treated as immutable, but that affects the frontend much more than the backend)
I should also point out that although the top level folders aren't "domains", this is still domain driven design. We've just grouped the high level concerns of our domains first. You could flip this over and have domain/infrastructure/method.ts, and I have tried it this way around, but you will almost definitely hit cross-domain issues that don't exist in this format.

Conclusion

So that is an extremely long (but honestly brief) tour of my backend architecture. It's quite a lot to wrap your head around but I have confidence (and experience) that it is an extremely clean, testable, scalable application structure.

What's in a tech stack?

Jack — Thu, 20 May 2021 14:08:24 +0000

What's in a tech stack? We always talk about MEAN, MERN, MEVN, LAMP, but honestly I've never understood the point of these acronyms.

I've been a dedicated front end engineer for 5 of my 8 years of professional development. So I've really only ever cared about one letter of these acronyms (and for LAMP I actually have zero investment). Do I care what language the backend is written in or what database we're using? I really don't! I'm using angular/react/vue/etc. and as long as I get a well formatted response from the backend I don't care.

As a full stack developer, or when working on personal projects, I might have more cause to describe the stack. But I've never described myself as working on a SAC stack (SQL Server, Angular, C#), or a JERN stack (Java, Express, React, Node) (I don't even know what database we used for that one, I was so far removed from the backend it just didn't affect me), or a JRPK stack (figure that out if you can).

The other problem with these acronyms is they don't tell the full story of the stack. Why do we explicitly call out Express in the MEAN stack? Is it because it's more fundamental than any other library/framework we're using? I think it's more likely that MAN is just less catchy than MEAN.

So on that note, what stack am I using for my application? I'm actually owning the entire stack, so for once the entire tech stack actually does matter to me. Rather than try to create a pointless acronym I'll just list out some of the technologies/frameworks/libraries I'm using and why:

react

I've used Angular.js, Vue, and React over the years. And although I've always had a soft spot for Vue, the more I use both Vue and React, the more I find React to be the more complete/powerful option.

tailwind

I've used many languages in the past; sass, less, react-css, linaria, I even wrote plain css one time. But in all honestly, I hate writing css - which is probably a weird thing for a front ent dev to say. There's just something really cumbersome and unmaintainable about it for me. Tailwind offers just the right amount of abstraction between writing css and using something like bootstrap.

It's quite a divisive library, but almost every complaint I see about tailwind is solved by writing small components that remove the classes from my main application code. The components themselves give elements semantic meaning, not the class names - that's the secret.

Most importantly for me, tailwind makes prototyping and rapid development so easy.

react-hook-form

I've not been using this for long but it does the job nicely and it certainly feels like it performs better than formik and redux-form.

I felt its inclusion in this list was necessary because forms are like 80% of almost every front end application.

jpex

I can't write an article without plugging jpex of course, so here it is. Dependency injection is a pretty fundamental part of my architecture which I'll cover another time. I use jpex in both the frontend and the backend applications and it is essentially the glue that ties everything together.

respite

This is a state management solution I built myself. It's similar to react-query, I could have used react-query, and I have used it extensively in the past. But there were a lot of inconveniences and strange decisions that would constantly irritate me.

I also wanted a way to do global state that didn't involve adding another large library, as well as a way to decouple cache invalidation from the mutations.

I don't expect anyone else to use respite, but the more I work with it the more I'm pleased I wrote it! I'll probably cover some of it at a later point.

vite

Worth covering because 99% of apps are built with webpack. Webpack just owns the world of ui building. But I've always felt there are better ways of doing things. I originally played with snowpack when it was still in beta, but when vite came out by the guys behind Vue, I knew it would have the same super-simple api and great docs so I took the plunge.

I love it and I won't be willingly going back to webpack in the future!

node

Node was an obvious choice as it means my entire app is written in the same language. I'm always surprised just how fast it runs too. I've used node for all sorts of cli tools and bits and pieces for many years, so it's always my go-to for most tools and apis.

express

I actually tried out a few alternatives to express but nothing really beats it. It baffles me how a library can just "get it right" first time! Express has been the defacto server library for... ever!

mongo

To be honest, I haven't 100% settled on a database for production. But during development mongo is an obvious choice. It's so simple to set up and use. And you can just start throwing data in it without having to spec out an entire schema or relational mappings.

typescript

Typescript is another absolute fundamental part of the stack. I don't think I could go back to plain javascript at this point in my career. If I ever have to write javascript I immediately feel uncomfortable "What type is this variable meant to be? What parameters am I meant to pass into this function?!" - so stressful.

So there you have it. And of course this doesn't tell the full story. I've tried to include the fundamentals, but there are so many baselines that we just take for granted. Babel, jest, testing-library, these are all just as important as, say, express.

I'm sure you're wondering where next.js is in this list, everybody uses next.js now right? Well not me! I'm a rebel and I will probably cover my reasons why in a future post!

Okay so officially this is the MERN stack I'm using. But that tells you nothing about my actual application at all. Personally I prefer to call this the MERRVJNTT stack.

Building a serious(ish) project - intro

Jack — Thu, 20 May 2021 13:55:38 +0000

I've been a developer for 8+ years now, working in all kinds of industries. Some full stack jobs, mostly front end jobs. I've created ~35 open source libraries over the years. I'm technically self-taught but I've also learnt a lot from some pretty smart people I've had the fortune to work with.

I've finally committed to building a full stack application. I wanted to get some of my thoughts, ideas, and headaches written down as I go through this process. There are a lot of things I've picked up in my career that I just store internally, so hopefully this will force me to start sharing things a bit more!

There will also almost definitely be a lot of things I've never had to deal with before, and things I'll have to learn from scratch. So I also figure it might be good to share these experiences too.

This is probably the first time I've been this excited to make something, and (outside of my actual job) actually want to see it through to the end. Let's see what happens...

Don't mock modules

Jack — Tue, 04 May 2021 10:27:25 +0000

I want to talk about a practice that has more or less become the defacto for javascript unit testing: mocking module imports.

I have been wary of this from when it first came about with rewire and similar libraries. As time has gone on my opinion has only strengthened, and with jest actively embracing mocking modules, I feel like nobody has even bothered to make a counter argument.

So, this is my argument against mocking module imports.

What is a mocked import?

When you're writing code, you will at some point need to access some external resource. Fetching data over the network, reading a file from disk, etc.

When you're trying to test your code, you really don't want to be using these external resources. Imagine if your tests had to manage reading and writing files. You'd have to write the initial file state, run the test, tear down the file, hope there aren't any write errors or permission issues. Your tests would be slow, extremely brittle, difficult to run in a CI environment, impossible to run concurrently, and so on.

So we dont want to read or write real files during our tests. Ideally we want a fake fs module we can mock and assert against.

The idea of mocking imports is that you can patch node's require function, intercept certain calls, and potentially provide an entirely different module.

Once this idea was formed, it took off and this is what we have basically done ever since. It does the job, right? I can mock the fs module in my test, and then my code will get the mock when it tries to import it. I can spy on it, I can return a mocked response from fs.readFile. So what's the problem?

Tightly coupled to node

Module mocking is literally a monkey patch over node's require function. It doesn't operate on documented features of node. It is, essentially, a hack. There is no guarantee that the node implementation will always be compatible with the way module mocking currently works. In fact I would highly suspect that node releases have caused module mocking libraries to fundamentally break in the past.

We now also have native esmodule support in most environments, including node. But this is not how esmodules are meant to be used, they weren't created with module mocking in mind.

Sometimes monkey patches and hacks are a necessary evil in life, but it shouldn't form the basis of every test we write... should it?

Tightly coupled to implementations

Perhaps this is subjective, but I strongly believe a low level action like reading/writing should be kept as far away as possible from high level code. In other words: in high level code, low level actions should be abstracted.

Say you're fetching an auth token and you want to store it. It's so easy to just import cookies from 'browser-cookies', call cookies.set and you're done. I mean this is why javascript is so flexible right?

But should your auth function really know about cookies? What if you decide you'd prefer local or session storage? Or you even decide to just keep the token in memory. Abstraction is key to clean elegant code.

What does this have to do with mocking modules? It doesn't force tight coupling directly, but it does encourage this. Or perhaps it's more appropriate to say it encourages laziness.

Instead of asking "how do I separate my low level implementation detail from my application?", considering structure, maintainability, and coding practices; it's too easy to just go "ah well let's just mock the entire module import and move on".

Leaky Tests

When you mock a module, you're mocking the module for that entire test suite, potentially the entire test run, depending on your test runner.

How many times have you had a test fail because of another test? We're now adding even more global side effects into the mix.

What if every test requires a completely different response from your mocked module? What if you only want to mock part of the module? What if you want to completely undo a module mock for a single test? There are workarounds for these things, of course, but they are all workarounds and hacks, and quite often create more issues than they solve.

Side effects

For me, all of these points really boil down to a single fundamental problem: side effects - both intended and unintended.

Using the fs module directly will cause side effects in your code. Mocking the fs module will cause side effects in your tests. Mocking global objects like window.matchMedia comes with the same issues as module mocking.

Solution

I believe the solution is one that is a fundamental part of almost all languages except javascript: Dependency Inversion.

I'm not even suggesting you use some high level ioc container framework or injection library here. Basic dependency inversion can be done without any complexity or overhead!

You want to read from a file? Provide fs as a parameter. Want to check window.matchMedia in your react component? Create a context provider for it - the context api is dependency inversion!

When you start inverting your dependencies, unit testing becomes so much easier! And for integration/e2e testing you can just leave dependencies in place.

There are libraries/frameworks out there to give you a full DI experience. I'd highly recommend at least trying one out. I have a bias, of course, for my own DI library but this is not a plugging article, I just want to bring attention to the fact that there is a very simple and easy solution to the issues caused by module mocking!

The slightly misleading terminology of react hooks

Jack — Fri, 05 Feb 2021 23:08:32 +0000

So react hooks are awesome, they make components much simpler, they let you abstract your component logic and completely do away with class components.

However; the names of these hooks are somewhat misleading. Especially after react went all functional on us with function components.

Allow me to elaborate:

Side Effects

What is a side effect? A side effect is when a function changes another part of an application outside of its own scope:

function goHome() {
  window.location.href = '/home';
}

Calling this function is updating the window location, which is a global variable and isn't part of its own scope, so it's a side effect.

It's also worth quickly also covering side effect's partner-in-crime:

Idempotent

An idempotent function will return the same value for the same set of inputs every time.

function getPath() {
  return window.location.pathname;
}

This function is not idempotent. If the location were to change, you would get a different result.

With that out of the way, let's talk about useEffect:

useEffect

Accepts a function that contains imperative, possibly effectful code.

useEffect(() => {
  window.location.href = '/home';
}, []);

The useEffect hook allows you to do effectful behaviour inside a functional component.

If you do a side effect inside useEffect, react pushes that side effect to the end of the render cycle. In react terms, useEffect is about protecting the render cycle, not about removing side effects.

Why is this distinction important? We often talk about react components as being "pure". A pure component should be side-effect-free and idempotent. Wrapping your side effects in useEffect does not mean your component is still pure.

useState

I've seen many people say things like "we don't need smart/dumb components now we have hooks"

I don't want to go in-depth into patterns and architecture, but just because you're using useState instead of redux, etc, you're still introducing state into a component. This breaks a component's idempotency.

For every piece of state a component manages, the harder it is to reason about, test, and control that component.

Tl;dr useState is no different to any other component state - it is not functional.

Function vs Functional component

Okay something that I only really noticed recently: we always talk about functional components but actually, the docs refer to these as function components. What's the difference?

I believe the difference is this:

A functional component adheres to the rules of Functional Programming. It is idempotent, pure, and stateless.

A function component is just any old component that just happens to be a function.

That's a pretty big discrepancy. When we're talking about function components, do we care about whether it's a functional component?

I used to believe that yes; any function component should be side-effect and state free. Now I'm not so sure.

Conclusion

We've definitely got some confusing terminology here. useEffect is not offering some way of keeping effects "out" of your component. And a function component is not the same as a functional component.

Should I stop using hooks?

Absolutely not. Hooks are awesome. I just want us to be mindful that hooks are not some magical construct that preserves function purity. If your component has state or causes side effects, useState and useEffect do not change this fact.

Object.entries is awesome

Jack — Fri, 05 Feb 2021 19:01:25 +0000

Somebody asks you to loop through the properties of an object, most of us will reach for Object.keys right?

Object.keys(obj).forEach(key => {
  const value = obj[key];
  // do something
})

Somebody asks you to map the properties of an object to other values. Object.keys to the rescue again right?

const newObj = {};
Object.keys(obj).forEach(key => {
  newObj[key] = fn(obj[key]);
});

Wait, no reduce? That's so 2018! Let's be needlessly functional:

const newObj = Object.keys(obj).reduce((acc, key) => {
  return {
    ...acc,
    [key]: fn(obj[key])
}, {});

Nice!

Thing is, Object.entries and Object.fromEntries are now widely supported. You can now achieve the same thing like this:

const newObj = Object.fromEntries(
  Object.entries(obj).map([key, value]) => {
    return [key, fn(value)];
  })
);

It's also trivial to just make a utility function that combines the fromEntries and entries calls:

const mapEntries = (obj, fn) => Object.fromEntries(
  Object.entries(obj).map(fn)
);

you could then write the above like this:

const newObj = mapEntries(obj, ([key, value]) => [key, fn(value)]);

I love how simple this is, and you can do so much stuff that's always been a bit of a pain before.

Want to transform keys?

const newObj = mapEntries(obj, [key, value]) => [fn(key), value]);

Want to invert an object?

const inverted = mapEntries(obj, ([key, value]) => [value, key]);

Want to filter properties out of an object?

const filtered = Object.fromEntries(
  Object.entries(obj).filter(([ , value]) => {
    return value === true;
  })
);

Object.entries is awesome, have a play with it.

Case Driven Testing

Jack — Wed, 27 Jan 2021 20:35:08 +0000

I've wanted to write about testing for a long time - front end testing has been a bugbear of mine for years. But my opinions have changed constantly until recently.

I used to think "everything must have a unit test!" to the extent that I wrote unit tests for code that didn't even do anything. I wrote tests for dummy data even! But the point was that I didn't really give integration or end-to-end testing a second thought for a long time.

Then I started paying more attention to the testing pyramid, and it became a pointless balancing act of quantifying my tests. "This piece of code has 3 integration tests therefore it should have at least 15 unit tests and one e2e test" Again at the time this seemed perfectly reasonable - perhaps I couldn't see the wood for the trees.

Then I started thinking in terms of the testing trophy, where we think more in terms of how much effort should be given to each area, or how much confidence it gives you. In my experience it leads most people to write almost exclusively integration tests.

But recently I took a step back and it became a lot clearer than trophies and pyramids and coverage reports etc. Rather than setting out to write a certain number of a certain type of tests for a certain piece of code, you should examine the code and then think to yourself "what kind of test do I need?"

What kind of test do I need?

Branches

The main criteria I have for any piece of code is "does it have several branches?". If so, then I should attempt to write unit tests. Usually integration and e2e tests won't cover every if statement or error path, so a unit test is the best way to cover all branching options.

Dependencies

The second criteria I have is "how many other pieces of code use this piece of code?". If the answer is 1 then I consider whether it's worth testing it in isolation or if I can test it by testing the other code. For more than 1 we need to consider whether the code's coverage will potentially get lost.

As an example, let's say we have a ProjectDropdown component. It's pretty simple, has some integration with your form library, has a hardcoded list of projects to pick from, updates the form when you select a value, etc.
However, this component is only used by your ProjectForm component. So when we write tests for the ProjectForm, will we naturally be covering all of ProjectDropdown's usecases? If so then we'll probably just be repeating ourselves. We can probably get away with just testing the form instead.
Now, if ProjectDropdown has some different branches that won't 100% be encountered by ProjectForm's tests, then yes, it probably needs its own tests.

Confidence

My third criteria is "does a unit/integration test tell me anything about this piece of code?" Usually this is in terms of confidence I suppose.

The example

When I'm writing a component I usually split it into several pieces: a dumb component, a smart component, and the api layer.

So the dumb component probably wants a high level unit test. We want to confirm that the expected content is available, that interacting with elements emits the expected events, that the edge cases and paths are all handled.

The word "unit" is a bit muddy for components as you're usually writing your tests from a user's perspective, but you should also be trying to isolate your component as much possible

For the api layer, how much stuff does it do? Does it have much branching? For 90% of the time the api layer will be a single function that accepts some args, makes a fetch request, and returns the response. So all a unit test would tell us is that we're sending a request and we're returning the response. We could write an integration test but will that tell us anything more than an integration of the smart component?

You get the idea. A smart component would then have some integration tests that verify that it renders the dumb component, and it sends data to the api, and renders something else with the response data.

And then...

Another point I should make is that it's difficult to follow this method and adhere to TDD, as you have to have some understanding of your code before you decide what kind of tests to write. I find the key is to write some simple specs first that outline the general behaviour of the feature. You can refine and iterate on your tests as the implementation develops.

Do we need another state management solution?

Jack — Thu, 31 Dec 2020 17:30:24 +0000

So I spent several years working with redux + rxjs. Honestly epics and streams add so much cognitive complexity that it hugely affected the learning curve for new starters.

We then completely scrapped redux in favour of react-query. I've used it in enterprise apps for a year and I loved it. But it still didn't quite do what I wanted. I've yet to find a situation where I want to arbitrarily refetch my data, or refetch on window focus. Maybe for a chat app or something with constantly changing data. I also wish it was lazier, and smaller, and had better refetching logic.

So recently I started playing with recoil. But what starts as a very react-ey api quickly becomes convoluted and hard to read. And I also couldn't get past the whole "selectors should be idempotent" while also showing me a selector that reads from a database!

I also started to play with relay but I quickly abandoned it as the api just felt really awkward to me.

So like every npm package ever, I think "well this does 70% of what I want it to do". And like half of every npm package ever, I think "maybe I'll try writing my own version!"

I started playing with a proof of concept, using it in personal projects, refining it, iterating on it, and I've actually reached a point where I genuinely enjoy using it more than other libraries.

What do I do now? Do I publish it? Should I bloat npm even further with something that is really quite similar to existing products? Can I compete with Facebook et al? Do I even want to? Does the world really need yet another state management solution?

That's not rhetorical, I'd genuinely like your thoughts on this!