DEV Community: Jono M

Joys and woes of monorepos

Jono M — Fri, 17 Nov 2023 21:03:55 +0000

Monorepos are a great concept, especially in environments like Node.js which encourage having many small packages.

There are some very obvious advantages to this: not having to have a bunch of folders open and not having to push to seventeen different repos when you update a shared library is very nice. There are also some less obvious advantages, and as you get deeper into monorepos certain disadvantages start rearing their ugly heads as well. These are just a few good and bad points I've found in my own work that might be worth thinking about.

I've mainly used Yarn 1 workspaces as monorepo tooling so the benefits and problems may not apply to other tools, but I assume the underlying ideas are similar and I'm curious to try out other tools to compare.

Standardised tooling

One of the greatest things about a monorepo is that you can centralise a lot of tooling configuration. For example, chuck a .prettierrc in the root of your monorepo and you'll never have arguments about code style again. Put a few base tsconfigs in the there to extend, and suddenly TypeScript behaves with beautiful consistency across all your packages. Add formatting, linting and husky as dependencies in the root of your workspace and you no longer need to work through upgrading everything individually when a new version comes out.

This is especially true in a team, and even more especially true if you often spin up new packages. Keeping the tooling consistent makes it vastly easier to jump back and forth between packages your team and other teams have worked on.

Dependencies within the repo just work

Something I discovered embarrassingly recently is that if you have a monorepo package that depends on other packages in that repo, and you don't need to publish it (for example, it's a frontend app you bundle anyway), you can add non-published packages as dependencies and it just works.

Let's say you have a design system and several frontend apps which depend on it. Those frontend apps are just bundled into single JS files and deployed somewhere, so they don't need to be published. That means you can just put your design system somewhere in the monorepo with "name": "my-design-system" in its package.json and then add "my-design-system": "*" to any packages which depend on it - no need to deal with any package registries. Run yarn and you're set.

The way this works is that Yarn will symlink all your workspace packages into node_modules automatically, so they'll be found by the module resolution algorithm anywhere within your monorepo. A very neat idea.

This does however mean that if you try to publish a package that depends on a non-published package everything will work fine in CI because it's in the monorepo, but it'll break horribly on NPM because that non-published dependency can't be found. Bear that in mind!

Hoisting dependencies can cause problems

This is a thorny enough issue that I wrote a separate article about it!

The editor conundrum

Monorepos can grow to be massive and can contain a large number of disparate projects, which raises a conundrum: opening the whole monorepo in your editor makes finding files tricky because there are simply so many, but opening subfolders individually can kill many of the advantages of using a monorepo in the first place.

This is of course a problem with all large projects, but it's especially true of monorepos since their point is to become large and somewhat heterogeneous.

I guess my overall point is that monorepos should be managed with care: they're a great idea and often the only reasonable way of managing projects in environments like Node, but diving in without being aware of the pitfalls can end up causing you endless pain with everything from debugging dependencies to navigating the code.

Best of luck with your own monorepo journey!

Dependency hoisting in Yarn monorepos

Jono M — Fri, 17 Nov 2023 21:02:52 +0000

I ran into a strange problem the other day which demonstrates one of the problems with monorepos, at least when using Yarn 1.

I noticed that adding @testing-library/react to one of our packages caused tests in another unrelated package to suddenly start failing. This turned out to be a weird quirk of how Yarn "hoists" dependencies.

As a general rule, if you only have one version of a dependency in a workspace, Yarn will put it in node_modules at the top level. This avoids duplicating packages unnecessarily. However, sometimes different packages will depend on different versions of a library, in which case Yarn will put the version used by the majority of packages at the top level and create separate node_modules folders in the individual package directories for packages which depend on a different version.

The cause of the failing tests turned out to be this strange issue:

In our monorepo, we have several packages built in React 18 and several built in React 17
This means we have packages depending on @testing-library/react versions 14 and 12, which are compatible with the respective React versions
We had more packages using React 18, but more packages using @testing-library/react version 12 (compatible with React 17)

This meant that we ended up with directories like this:

node_modules/
  @testing-library/
    react/ -- v12
      node_modules/
        react/ -- *v17, because the top level package is 18*

  react/ -- v18

packages/
  some-unrelated-package/
    node_modules/
      react/ -- v17

You'll notice there are two copies of React 17 here - one because the package itself depends on React, and one because the testing library at the top level depends on a different version of React than the top level one.

The end result was that both versions of React were loaded in tests (one by the code and one by the testing library), which of course got React super confused and caused a bunch of strange errors.

Solutions

The best solution here would obviously be to upgrade everything to the same version of React. In fact, I'd say the moral of this story is to keep your dependencies in sync as much as possible in monorepos to avoid this kind of problem from the start.

But a couple of workarounds I found were

Just test more packages. If every package with React is tested with that library, the number of packages depending on each version matches and the right versions get hoisted
Use the nohoist property in package.json to force @testing-library/react not to be put at the top level. This means there will always be a copy per package, which will always use the nearest react (i.e. the same one as the package itself)

This is one of the trickier pitfalls I've run into with monorepos, and one you should absolutely try to avoid!

Functional immutable game state

Jono M — Tue, 17 Jan 2023 04:31:45 +0000

All-Seeing Wizards is a fun little game project I’ve been coming back to occasionally for some years now. I was originally inspired by Jackbox Games and thought I’d have a go at making that style of party game, with one big screen to show the board and players joining on their phones. I've struggled to get into some of Jackbox's games in the same way as a group board game though, so I was aiming to create something in that Jackbox style but with more traditional card-based and turn-based gameplay.

I'm not an artist, but pulled together enough simple assets for a proof-of-concept!

The overall idea

The core concept of the game is that the players play action cards to move around the map, attack each other, defend, and so on. The catch is that each turn the players secretly choose 4 action cards per turn, but then everyone's first card activates together, followed by everyone's second card, etc. This means that in order to win you need to predict where your opponents will move this turn, when they will attack, and so on.

(there's a bit more about how to play on GitHub)

This also means that it's important to be able to "preview" a set of actions in the player view while people are choosing cards, then gradually apply those actions to the game state and animate them in the host view so everyone understands what just happened.

A player in the middle of picking cards sees a semi-transparent preview of their actions

The issue with this is that game programming traditionally uses mutable state and imperative programming, meaning the game state is updated in place each time it changes and there's no easy way to "preview" and "revert" updates. There are of course ways around this, but I like functional programming and this seemed like an interesting place to try a non-traditional approach with immutable state handling.

Immutable basics

Enter Immutable.js, a library for creating and updating immutable state in JavaScript. With usual mutable JavaScript objects you might write something like:

const gameState = {
  players: [{ character: 'darkLord', hp: 5 }]
}

gameState.players[0].hp = 3

With Immutable.js you might write it more like:

import { fromJS } from 'immutable'

const gameState = fromJS({
  players: [{ character: 'darkLord', hp: 5 }]
})

const newGameState = gameState.setIn(['players', 0, 'hp'], 3)

Notice that in the first example the original game state was lost, but in the second example we now have both the original and modified game state.

This approach is more performant than it looks; Immutable.js will reuse existing objects wherever possible, so rather than creating a clone of the entire state, it only clones the objects needed to express the new properties. If you have 4 players and modify one of them, the same object references will be returned for the other three.

The Immutable.js README has a much more complete description of immutability and why you might want to use the library. Also worth mentioning that Immer is an alternative which is a bit easier to get started with.

In the game

The way this works in practice is a bit more complex, but follows the same idea. The core piece of logic is a function called performTurn, which takes a game state including players who have each picked cards to play that turn, and returns the new game state after the turn is complete (plus a list of what actions happened during the turn).

In vastly simplified terms the function looks like this:

game_state = original_game_state

for i in 1 to 4:
  game_state = apply cards placed in slot i

game_state = discard picked cards and draw next hands

return game_state

The important thing about this is that it's a pure function; it doesn't change the original game state and just returns a new altered state. This means I can easily do all the following

Display the current game state on the player and host screens
Overlay the expected movements and attacks on the player screen when they pick cards, by calling performTurn with the cards they picked
Remove that overlay when they deselect cards
Immediately compute and persist the new game state on the server once all players have picked cards (to maintain data consistency - the server could crash at any point and the game would always be in a state where you can continue playing)
Gradually compute and display the new game state on the host display once the server has updated

Because all the functions involved are pure and idempotent (the same input will always result in the same output), the server and client can trust that the same actions applied to the same initial state will definitely get the same results, so each client can safely apply the actions without needing constant confirmation from the server.

The host display plays through all players' actions and shows their effects

Side note: Shared logic

This setup is doubly convenient because the core game logic is actually shared between the server and client. Since the whole project is built in TypeScript, I realised that rather than having the server and client constantly communicate about game states (increasing latency and network usage) I could just put the game logic in a shared folder and import it into both the server and client code.

Obviously it's still important to make sure the server has the "master" copy of the data and prevent clients from making arbitrary modifications, but with a clearly defined websocket interface I think overall this has the best of both worlds.

Inspiration

This project setup was inspired by a screencast by Gary Bernhardt (of "Wat" fame) called "Functional Core, Imperative Shell". This sounded hard to implement in most production applications but a great idea in concept, so I wanted to try it in a smaller project which already has a conveniently clear boundary between the core (game logic) and shell (client, network and database handling).

By the way, I can highly recommend his Destroy All Software screencasts in general - as well as specific techniques they have a ton of valuable insights for how to think about code.

And finally

My general point here is that there's always more than one way to approach a problem in any field, and sometimes you can get surprising advantages from going against whatever approach is standard in that field. Imperative and functional programming each have their tradeoffs, and each sometimes finds value in unlikely places.

Yet another post about SolidJS vs React

Jono M — Thu, 12 Jan 2023 21:11:47 +0000

I've recently been working on a little personal finance app as a way of learning and experimenting with Bun and SolidJS.

I originally built it in RedwoodJS, but wasn't super happy with the TypeScript integration and found React a bit slow and hard to optimise on mobile. (Yes, it was probably my fault for writing a slow UI. But hey, it's a personal project, so sometimes it's more fun to try a different framework than optimise the current one.)

In the JavaScript ecosystem a new framework comes out every other day, so why bother with Solid? Nothing the framework does felt super revolutionary to me - Svelte already had a "compile JSX to DOM operations" approach, Vue already had an observable reactivity model, and there are many frameworks that look like React but are smaller and/or perform better. Where Solid stood out was in combining those into a simple(ish - we'll get to that), coherent and minimal-but-fully-featured package.

The cost of reactivity

If you're coming from React, the way Solid works takes a bit of wrapping your head around. But once you figure it out you have a moment of epiphany and wonder "why doesn't React work like this?"

Let's look at a simple example to illustrate this. In React you might write:

const Counter = () => {
  const [number, setNumber] = useState(0)
  const squared = number * number

  return (
    <button type="button" onClick={() => setNumber(number + 1)}>
      {number} squared is {squared}
    </button>
  )
}

A couple of things about this are suboptimal.

Each time number gets updated, the whole function body is re-executed. This won't be noticeable now, but when you have a big component tree and something changes at the top, it can be tricky to avoid those extra milliseconds executing every function in the tree
Re-executing the function isn't the only work that will be done when number is updated. React will also build a whole tree of JavaScript objects for the elements on the page and use its Virtual DOM to create a diff and apply the updates to the page
There's a subtle issue with the onClick handler - because we create a new function every time this component is rendered, it will remove and re-add an event listener on the element every time. This is yet more unnecessary work. We can avoid that with useCallback, but in my experience that trick is not obvious or easy to remember for most developers

Let's try converting this to Solid then. Solid has createSignal which on the surface works pretty similarly to setState, so we can naively try this:

const Counter = () => {
  const [number, setNumber] = createSignal(0)
  const squared = number * number

  return (
    <button type="button" onClick={() => setNumber(number + 1)}>
      {number} squared is {squared}
    </button>
  )
}

Running it gives us:

Well that didn't work. One key difference between Solid's createSignal and React's useState is that when we reference the state we need to use a function call, i.e. number() instead of number. This is because it's not just a number; it's an observable reactive value, which means when it changes Solid can detect the change and work out which exact parts of the page need to be updated with the new value.

So let's find and replace number with number() and see what happens.

That's a lot closer, but something is still not right. It's now incrementing the number correctly, but squared is not responding.

Let's extrapolate the same rule: using reactive values should look like function calls. So what if we make squared a function as well? Let's try squared = () => number() * number()

Success! Now we have the same reactive UI, written in a style which is subtly different but recognisably the same idea:

const Counter = () => {
  const [number, setNumber] = createSignal(0)
  const squared = () => number() * number()

  return (
    <button type="button" onClick={() => setNumber(number() + 1)}>
      {number()} squared is {squared()}
    </button>
  )
}

Try it on Solid Playground

In comparison to the three issues with React above:

The function body is only executed once
- We know this because squared didn't update in our second attempt
When number is updated, only the text in the element will be updated; there's no big JavaScript object tree and expensive diffing calculation
- We can sort of see this by looking at the generated output code on Solid Playground. There are a couple of calls to Solid's internal insert function, which basically watches a reactive value and does a single DOM update when it changes
The event handler is only created once
- We can see this in the output on Solid Playground as well. There's a line looking like _el$.$$click = () => setNumber(number() + 1), and reading through the code around it we can see that it's in a function that's created and then immediately executed once

So Solid's reactivity works something like this:

Some Solid functions like createSignal return special reactive values
We can get the current value with a function call, and any UI which depends on that will be reactively updated
To compute values while maintaining reactivity, we need to create functions; then the function will be automatically re-executed when the underlying reactive state changes

Cool knock-on effects

This reactivity system also makes side effects and global state much nicer.

To do side effects, in React you might write something like:

useEffect(() => {
  console.log('number was updated to', number)
}, [number])

But because Solid can watch reactive values and re-run functions when they change, in Solid you can just write:

createEffect(() => console.log('number was updated to', number())

No need to specify a list of dependencies and add ESLint plugins to make sure you don't forget any. Just use those reactive values and you're done.

For global state, in React you probably need a helper function from a library like createGlobalState from react-use.

But in Solid, you can just call createSignal at the top level to create global reactive state - it's convention to call it in components but it works the same anywhere.

That seems so simple and consistent!

Well ... yes and no.

This function-call-style essentially exists to get around the limitations of JavaScript's primitive values (numbers, booleans, and so on). Once you start dealing with objects, constructs like Proxies exist which can fulfil the same purpose entirely transparently.

Because of this, props and stores (more complex state management) break this function call convention and just look like regular objects:

const Numberer: Component<{ count: number }> = (props) => {
  const [myContrivedStore, setStore] = createStore<{
    oneLess: number
    oneMore: number
  }>({
    oneLess: props.count - 1,
    oneMore: props.count + 1
  })

  createEffect(() => {
    setStore('oneLess', props.count - 1)
    setStore('oneMore', props.count + 1)
  })

  return (
    <div>
      One less: {myContrivedStore.oneLess},
      one more: {myContrivedStore.oneMore}
    </div>
  )
}

Either of these systems would be fine on their own, but in combination they can get confusing. It also means that instead of natively destructuring props you need to use the somewhat awkward splitProps helper function.

To be honest I wish everything just consistently used function calls, but that has its own drawbacks such as becoming increasingly awkward and heavy with deeply nested objects. I wonder if it would be possible to build an editor plugin that detects and highlights which variables are reactive (JavaScript is too dynamic to do that perfectly, but with TypeScript it ought to be possible to highlight typed variables at least).

Anyway, all this means Solid's reactivity system takes a bit of wrapping your head around - you need to get used to remembering which variables are reactive vs regular objects - but once you get the hang of it the framework feels more natural and expressive than React while also generally being only a few percentage points slower than vanilla JS.

Is React bad then?

No, of course not. There are huge numbers of companies using React in production and their apps work just fine. There's also a giant ecosystem around React, while Solid is still relatively young and doesn't have the same number of supporting libraries and frameworks.

But Solid does seem to be gathering momentum (at least looking at GitHub stars!) and it might just be in the sweet spot of "simple enough and similar enough to React to learn easily, but different enough to be worth learning".

I can't say whether Solid is the right choice for your next giant enterprise project, but it's something to keep an eye on and I'm definitely rooting for it to succeed.

Adventures learning Python

Jono M — Wed, 11 Jan 2023 21:01:59 +0000

I've recently been looking at some open source Python projects and thinking I should give it a go again. I last seriously did Python during university, and since then I've basically been on the Ruby train. But reading through some open source code, I was pleasantly surprised by how readable Python is - enough that I thought I'd try and build a little project in it.

It's been quite a fun journey: Python is similar enough to my most used languages that it doesn't take a huge amount of effort to pick up, but different enough that the differences really stand out and take some thought and research to work around.

The good

Python has types now!

This was quite an exciting find, and made picking the language back up again a lot easier than I expected. It was especially nice to have auto-complete, auto-import and explanatory inline documentation for most of the libraries I added (yes, not all of those require a type system, but they're enhanced by having one).

The usability of the type system feels somewhere between TypeScript (awesome) and Ruby (underwhelming). It's nice to have built-in syntax for parameter and return types - doing this in Python:

def fn(x: str) -> str:
  ...

is much nicer than the equivalents in Ruby type checkers:

# Sorbet, struggling to coerce a type system into Ruby syntax
sig {params(x: String).returns(String)}
def fn(x)
  ...
end

# or RBS, putting the signature in a separate file
def fn: (x: String) -> String

...

def fn(x)
  ...
end

However, there are still some places where type checking feels tacked-on rather than a core feature of the language. It's weird to me that types like str and int can be used without imports, but the Any type requires from typing import Any. The idea of having an opt-in typing package like that is nice, but TypeScript seems to have a more sensible distinction between built-in and imported types.

Named parameters

I was ambivalent on named parameters last time I used Python, but my concern for readability has increased a lot since then and they're now one of my favourite features. JavaScript's explicit object syntax is awkward, Ruby's implementation with separate positional and named parameters better, Swift's internal and external names are fun but clunky, but Python's "any parameter can be used with a name" is a wonderfully simple and flexible implementation.

def listen(port: int = 3000, host: str = '127.0.0.1'):
  ...

# can be used like
listen(host='0.0.0.0') # easily override any parameter
listen(4000) # if it's easy to read, omit the name
listen(port=4000) # if it seems a bit unclear, add the name
# and many more combinations

Where in other languages you need to carefully think "when this is used, will the parameters be easy to understand?" while writing a function, in Python you just give the parameters sensible names and let whoever is using the function decide what will read best.

The 50/50

Imports

I remember when I used to like Ruby's require system where everything gets dumped into the global namespace. After working on some huge Rails projects, I'm now much more of a fan of JavaScript/TypeScript's import system which neatly isolates everything with none of Ruby's name clashes or weird namespacing errors. So I was happy to find that Python's imports work much the same way.

That said, I still don't really understand the import system. I especially don't understand why other projects I've seen can use from ..some.relative.looking.folder import ... and I can't. But I can use from some.absolute.path.in.my.project import ..., which seems more convenient anyway, so I don't know why some projects seem to avoid using that. I also find it a bit weird that imports use dots but effectively refer to folders and files.

So I guess I'm ambivalent. I like the isolation of Python's module system compared to Ruby, but it doesn't feel as immediately understandable as ES Modules - as a relative beginner it takes more background knowledge and context to work out how to create a models/__init__.py and then do from models import stuff compared to just creating models.js and do import stuff from './models.js'. But I have to assume Python's import rules are something you learn once and then aren't a big deal in practice.

The bad

Lambdas are awkward

I'm very used to doing nice functional data manipulation in JavaScript and Ruby, and it's sad to see that Python still can't do that level of expressive data manipulation. Where in Ruby I can do:

users_by_email = responses.map(&:user).group_by(&:email)

in Python I need to write out the lambdas with somewhat verbose syntax, import one of the functions, and write it "backwards" because these are functions rather than methods:

users_by_email = groupby(
  lambda user: user.email,
  map(lambda response: response.user, responses)
)

You can see how this would get complex with longer chains of methods. Obviously you can write this in different ways, but to me it makes the most sense to think in terms of small pieces of logic and visualise the data gradually changing as it flows through a pipeline, and expressing that is frustratingly awkward in Python.

No safe navigation

In web development there's often a lot of interaction with third party APIs where the response structure is ambiguous or changeable, and as far as I've found data access in Python is relatively awkward.

Unlike JavaScript's safe navigation syntax (response?.data?.user?.id) or Ruby's dig method (response.dig(:data, :user, :id)), Python's default dict access feels very limiting. You seem to end up doing a bunch of things like this:

data = response["data"] if "data" in response else None

# or

try:
    id = response["data"]["user"]["id"]
except KeyError:
    id = None

both of which feel long and complex compared to JavaScript and Ruby.

It's a similar story with objects: if you have an object which may or may not be present, there's no easy way to

Libraries like glom seem to alleviate the problem a bit for dicts and arrays, but the general lack of built-in null checking (or None checking, I suppose) feels worrying to me coming from the safety of TypeScript.

Procedural or OO?

One of my favourite things about Ruby is how consistent it is - "everything is an object" is a powerful concept which makes doing pretty much anything in Ruby feel nice and consistent with everything else.

Python doesn't have this, so for someone like me who isn't used to the language it feels confusing why some things are methods on objects and others are just functions. Why is list.count() a method but filter() is a function?

Overall

If you count things up, it seems I have more gripes about Python than things I enjoy. That doesn't tell the full story though; there are plenty of things I didn't mention because they're good enough to just work and stay in the background, which at the end of the day is how things should be in a language.

So although it's still early days, I have a more positive view of Python and I'm curious enough to keep looking for solutions or workarounds for some of those annoyances.