Scaling a react/redux codebase for multiple platforms

Eric Bower — Tue, 02 Feb 2021 03:12:34 +0000

In the world of react and redux, there is no shortage of tutorials, to-do apps,
and how-to guides for small web applications. There's a rather steep learning
curve when trying to deploy a modern web application and when researching how to
scale and maintain a large one, I found very little discussion on the subject.

Contrary to what people think, react is not a framework; it's a view library.
That is its strength and also its weakness. For people looking for a
batteries-included web framework to build a single-page application, react only
satisfies the V in MVC. For small, contained applications this is an incredible
ally. React and redux don't make any assumptions about how a codebase is
organized.

There is no standard for how to organize a react redux application.
We cannot even settle on a side-effects middleware for it.
This has left the react redux ecosystem fragmented. From
ducks to rails-style layer
organization, there is no official recommendation. This lack of standardization
is not because the problem has been ignored, in fact, the official redux site
states that
it ultimately doesn't matter how you lay out your code on disk.
In this article is to show how I like to build large accplications using react
and redux.

Inspiration

There really are not a lot of large and open codebases to gain inspiration from.
The most notable examples I have found are
Automattic's calypso and most
recently Keybase's client.

Uncle Bob's Clean Architecture argues that
architecture should describe intent and not implementation. The top-level source
code of a project should not look the same for every project.
Jaysoo's Organizing Redux Application
goes into the details of how to implement a react/redux application using a
feature-based folder organization.

Code Organization

Monorepo

On a recent project I was responsible for multiple platforms which include but
are not limited to: web (all major browsers), desktop (windows, mac, linux),
outlook plugin, chrome extension, and a salesforce app.

We decided that all that code should live under one repository. The most
important reason was for code sharing. I also felt it unnecessary and
unmaintainable to build seven separate repositories.

A quick overview

I leveraged yarn workspaces to
do all the installation. Every package was located under the packages folder.
Each platform had its own folder for customization under the platform folder.
Platform specific packages would also be located under the packages folder.
Although if desired it would be easy to move platform specific packages under
each platform folder respectively. This made initial setup easier to handle
because all packages lived in one place.

plaforms/
  web/
    webpack/
    index.js
    store.js
    packages.js
  cli/        # same structure as web
  salesforce/ # same structure as web
  desktop/    # same structure as web
  chrome/     # same structure as web
  outlook/    # same structure as web
packages/
  login/
    packages.json
    index.js
    action-creators.js
    action-types.js
    effects.js
    sagas.js
    reducers.js
    selectors.js
  logout/     # same structure as login
  messages/   # same structure as login
  web-login/  # same structure as login
  cli-login/  # same structure as login
packages.json

Feature-based folder organization

There are two predominate ways to organize code: layer-based and feature-based
folder organization. When building an application, the top level source code
should not look the same for every single application. The rails-style MVC
folder structure (layer-based) muddles each feature together into one
application instead of treating them as their own entities. Building a new
feature in isolation is more difficult when each component of a feature needs to
join the other features. Using a feature-based approach the new feature can be
built in isolation, away from everything else and then "hooked up" later when
it's finished.

Layer-based

src/
  models/
    login.js
    logout.js
  views/
    login.js
    logout.js
  controllers/
    login.js
    logout.js

Feature-based

src/
  login/
    model.js
    view.js
    controller.js
  logout/
    model.js
    view.js
    controller.js

Every feature is an npm package

This was a recent development that has been successful for us. We leveraged
yarn workspaces
to manage dependencies between features. By developing each feature as a
package, it allowed us to think of each feature as its own individual unit. It
really helps decouple a feature from a particular application or platform. Using
a layer-based approach, it's really easy to lose site that these features are
discrete contributions to an application.

Absolute imports

It was a nightmare moving code around when using relative imports for all of our
internal dependencies. The weight of each file being moved multiplies by the
number of thing depending on it. Absolute imports were a really great feature to
leverage. The larger the app, the more common it is to see absolute imports.

Lint rules around inter-dependencies

One of the best things about absolute imports was the lint tooling that could be
built. We used a namespace @company/<package> for our imports so it was
relatively easy to build lint rules around that consistent naming.

Strict package boundaries

This was another key to scaling a codebase. Each package had to subscribe to a
consistent API structure. It forces the developer to think about how packages
are interacting with each other and creates an environment where there is only
one API that each package is required to maintain.

For example, if we allowed any package to import another package, it's difficult
to understand what happens when a developer decides to move files, folders
around. For example when building a package, let's say we want to change the
file utils to helpers. By allowing a package to import utils directly, we
inadvertantly broke the API. Another example is when a package is really simple
and could be encapsulated inside one file. As long as the package has an
index.js file and it exports all of the components that another package needs,
it doesn't matter how the package is actually organized. It's important for a
large codebase to have some sort of internal consistency, however, I found
having some flexbility allows to fit an organization that matches the needs of
the feature.

Another reason why strict module boundaries is important is to simplify the
dependency tree. When reaching into a package to grab a submodule, the
dependency graph treats that submodule as a full-blown package. When creating
module boundaries and a pacakge imports another package, it imports the entire
package. This simplifies the dependency graph and makes it easier to understand.
Here's an article on the important of dependency graph.

Each package exports the following:

{
    reducers: Object,
    sagas: Object,
    actionCreators: Object,
    actionTypes: Object,
    selectors: Object,
    utils: Object,
}

Creating this consistent API provided opportunities ripe for tooling.

One of the most important rules was the module-boundary lint rule. This
prohibited any package from importing a sibling package's submodules directly.
They must always use the index.js file to get what they want.

For example:

// bad and a lint rule will prevent this
import { fetchNewsArticle } from '@company/news/action-creators';

// good
import { actionCreators } from '@company/news';
const { fetchNewsArticle } = actionCreators;

This setup came at a cost. Import statements became more verbose as a result of
this change.

Probably one of the greatest benefits to this structure was circular
dependencies. I know that sounds insane, who would actually want circular
dependencies in their codebase? Especially since every circular dependency that
was introduced caused an ominous runtime error: cannot find X of undefined.
I'll go into more details about why these errors were favorable later.

A package is a package is a package

Another huge benefit to our "feature-based, everything is an npm package" setup
was the fact that every package was setup the same way. When I onboard new
developers, I usually ask them to add a new feature. What this means is they get
to build their own package that does something new. This made them understand
exactly how a package works and they have plenty of examples on how to build
them. It really reduced the barrier to entry into a massive codebase and was a
great ally when trying to introduce people into a large codebase. With this
architecture, I created a scalable system that anyone can understand.

Support tools

Because of how tedious it can be to maintain a list of internal dependencies for
each package, not to mention creating package.json files for each feature, I
outsourced it to tooling. This was a lot easier than I originally thought.

I leveraged a javascript AST to detect all import statements that matched
@company/<package>. This built the list I needed for each package. Then all I
did was hook that script up to our test runner and it would fail a) anytime a
dependency was not inside the package.json or b) whenever there was a dependency
inside the package.json that was no longer detected in the code. I then built an
automatic fixer to update those package.json files that have changed.

Another huge benefit to having internal dependencies within each package was the
ability to quickly look at a package.json file and see all of its
dependencies. This allowed us to reflect on the dependency graph on a
per-package basis.

Making our packages npm install-able was easy after this and I don't have to do
anything to maintain those package.json files. Easy!

I wrote the support tools into a CLI
lint-workspaces

Package loader

Since I had a consistent API for all of our packages, each platform was able to
load whatever dependencies it needed upfront. Each package exported a reducers
object and a sagas object. Each platform then simply had to use one of our
helper functions to automatically load our reducers and sagas.

So inside each platform was a packages.js file which loaded all reducers and
sagas that were required by the platform and the packages it wanted to use.

By registering the packages, it made it very clear in each platform what kind of
state shape they required and what kind of sagas would be triggered.

// packages.js
import use from 'redux-package-loader';
import sagaCreator from 'redux-saga-creator';

const packages = use([
  require('@company/auth'),
  require('@company/news'),
  require('@company/payment'),
]); // `use` simply combines all package objects into one large object

const rootReducer = combineReducers(packages.reducers);
const rootSaga = sagaCreator(packages.sagas);
export { rootReducer, rootSaga };

// store.js
import { applyMiddleware, createStore } from 'redux';
import createSagaMiddleware from 'redux-saga';

export default ({ initState, rootReducer, rootSaga }) => {
  const sagaMiddleware = createSagaMiddleware();
  const store = createStore(
    rootReducer,
    initState,
    applyMiddleware(sagaMiddleware),
  );
  sagaMiddleware.run(rootSaga);

  return store;
};

// index.js
import { Provider } from 'react-redux';
import { render } from 'react-dom';

import createState from './store';
import { rootReducer, rootSaga } from './packages';
import App from './components/app';

const store = createState({ rootReducer, rootSaga });

render(
  <Provider store={store}>
    <App />
  </Prodiver>,
  document.body,
);

I have extracted the package loader code and moved it into its own npm package
redux-package-loader.

I also wrote a saga creator helper
redux-saga-creator

Circular dependencies

Circular dependencies were a very important signal when developing. Whenever I
came across a circular dependency, some feature was organized improperly. It was
a code smell, something I need to get around not by ignoring it, not by trying
to force the build system handle these nefarious errors, but by facing it head
on from an organizational point of view.

One of the 🔑 topics I learned about along the way was
Directed acyclic graph

I'll explain by example, give the following packages:

packages/
    mailbox/
    thread/
    message/

I would regularly run into situations where pieces of code within the mailbox
package would want to access functionality inside the thread package. This
would usually cause a circular dependency. Why? Mailboxes shouldn't need the
concept of a thread to function. However, thread needs to understand the
concept of a mailbox to function. This is where DAG came into play. I needed to
ensure that any piece of code inside mailbox that needed thread actually
didn't belong inside mailbox at all. A lot of the time what it really meant
was I should simply move that functionality into thread. Most of the time
making this change made a lot of sense from a dependency point of view, but also
an organizational one. When moving functionality into thread did not work or
make any sense, a third package was built that used both mailbox and thread.

Cannot find X of `undefined`

For whatever reason, the build system (webpack, babel) had no problem resolving
circular dependencies even though at runtime I would get this terribly vague
error cannot find X of 'undefined'. I would spend hours trying to track down
what was wrong because it was clear that this was a circular dependency issue.
Even when I knew it was a dependency issue, I didn't know what caused it. It was
a terrible developer experience and almost made me give up completely on strict
package boundary setup.

Tools to help detect them

Originally the tool that helped detect circular dependency was
madge. It was a script that I would run and it
would normally indicate what would be the dependency issue.

Once I moved to yarn workspaces however, this tool failed to work properly.
Thankfully, because every package had an up-to-date package.json file with all
inter-dependencies mapped out, it was trivial for to traverse those dependencies
to detect circular issues.

An open example

The project codebase is not publicly accessible but if you want to see some
version of it, you can go to my personal project
youhood. It is not a 1:1 clone of the
setup, primarily because I am using TypeScript for my personal project and yarn
workspaces was not necessary to accomplish what I wanted, but it organizes the
code in the exact same way by leveraging redux-package-loader.

It's not perfect

There are a few issues when developing an application like this.

Importing a package brings everything with it
Import statements are more verbose

In a follow up blog article I will go into more detail about these issues.

This code organization could build multiple platforms using most of the same
code. As with most things in life, this was not a silver bullet. They 🔑
take-aways were:

Feature-based organization scaled really well
A consistent package interface allowed for tooling
Force developers to think about dependency graph

DEV Community: Eric Bower

Scaling a react/redux codebase for multiple platforms

Inspiration

Code Organization

Monorepo

A quick overview

Feature-based folder organization

Every feature is an npm package

Absolute imports

Lint rules around inter-dependencies

Strict package boundaries

A package is a package is a package

Support tools

Package loader

Circular dependencies

Cannot find X of `undefined`

Tools to help detect them

An open example

It's not perfect

References

DEV Community: Eric Bower

Scaling a react/redux codebase for multiple platforms

Inspiration

Code Organization

Monorepo

A quick overview

Feature-based folder organization

Every feature is an npm package

Absolute imports

Lint rules around inter-dependencies

Strict package boundaries

A package is a package is a package

Support tools

Package loader

Circular dependencies

Cannot find X of undefined

Tools to help detect them

An open example

It's not perfect

References

Cannot find X of `undefined`