Git Submodules vs Monorepos

I am very interested in this topic and I want to know more about this. I have read about monorepos, and I understand the advantages, but I am more concerned about the disadvantages. More specifically, these four things:

1. In monorepos, how do you maintain a clean git history? For example, if you have backend + frontend in the monorepo, wouldn't the git history be a complete mess? Or you just don't mix these things?

2. Security. I have already seen that maybe this is impossible. If you want to restrict access to some "packages", then I believe this is imposibble with monorepos.

3. What happens when you have packages written in other programming languages other than Javascript? Can you still have them in the monorepo without conflicting with, for example, Yarn Workspaces or Lerna?

4. How do you run tests for only certain packages? How do you achieve CI/CD for certain packages?

Well, I find all of these things huge disadvantages of monorepos and I think these reasons are enough for not even trying them in real world projects. So trying to find a way to solve these problems, I thought about git submodules. To be honest I haven't worked too much with them (I barely know how to add them to a project and remove them).

Correct me if I am wrong, but with git submodules I think we do not have the capacity to generate "symlinks" like with monorepos. I was playing around with Yarn Workspaces, and when I make a change to one package, this change gets reflected in another package that uses it. This is cool! But I think this cannot be done with git submodules right?

I think that wit git submodules, we can say the following things about the points I just mentioned.

1. Independent repos have independent git histories, so I think this would get solved.

2. If I do not have access to one repo, then would running git clone --recurse-submodules just wouldn't give me those repos, or would it throw an error saying that the clone cannot be made because there are repos that I cannot access?

3. We can add submodules written in other languages. But I think they would be useful if, for example, I have some Flask app and the client in React, and if I make some change in the Flask app then I can test that change in the client immediately. Maybe this can be done with some kind of script that starts all the things inside each submodule? After

4. I think this point gets solved with independent repos.

I really need someone to clarify all of these things to me. I appreciate your opinions!

Comments

[Gahl Saraf]

This discussion is extremely one sided. There are many significant benefits to working with monorepos. Namely:

• Clear dependencies between different parts of the project. Even things like references from your IDE.
• Code reuse is much easier. Do you really want to spin off a small bit of shared code to yet another repo? Or put it in a shared repo with things semi-related? (in which case, isn't that one really a shared monorepo?)
• Much much simpler to make changes that span several systems. With multi-repos, you end up having to manage multiple PRs, and dependencies between them.
• Versioning is difficult or introduces overhead
• Splitting the repos can easily become difficult. Backend, frontend -> great. Shared API? Third repo. Another service? Fourth repo. Shared code between new service and backend? Fifth repo?? ...
• End to end tests are hard to get right - which versions of repos do they run on? What if there are dependent PRs?

Of course there are downsides too. The most significant one is the lack of good tooling.

Long blog post I wrote on this topic: raftt.io/post/development-challeng...

[Zenobius Jiricek]

It's also worth rebuking a claim that you should lock off or hide a portion of a project by making it a separate repo and using access controls for that repo.

This approach creates a huge problem for everyone in a company to understand how everything works. Not to mention that it becomes virtually impossible to spin up the entire ecosystem of your product on a local machine without requiring an extensive and standardised cicd pipeline which produces docker images.

If you just want to control the fact that changes to certain parts of the monorepo should always go throug ha codereview with a certain group of people, the use the CODEOWNERS feature of github.

The next rebuke is that after having experimented and worked with several monorepos over the last four years, i'll say these things:

• i'm not authoritive, and to be so is exhausting.
• and while i'm trying to avoid exhausting navel gazing exercise of bike shedding: just use NXDEV. NXDEV is less about nodejs and more about providing dependancy graph mapping tools. You're welcome to try and understand how BAZEL or PANTS work... but i noped out of them pretty quickly.

[Gahl Saraf]

I agree completely, though I think that if you are considering locking off parts of the codebase, you are likely far beyond the stage where the entire system can be run locally.

I've worked a lot with BUCK (FB/META's version of BAZEL; very similar semantics), and TBH I loved it. Haven't tried NXDEV, will take a look :)

[Fabio Gartenmann]

I think I got some news here, after having tried multiple monorepos (PNPM, Yarn, Turborepo, NX) then figured: I want every package standalone but while coding I want the comfort of a monorepo.

My thought

What if I add Husky pre-push hook to separate the packages and push them separately:

• Locally I get a monorepo
• Remotely the packages are split

so I‘ve searched for remote split monorepo…

The solution?

github.com/marketplace/actions/mon...

So at first glance it seems that using Github Actions solves everything in a very elegant way, using the CI pipeline…

• Monorepo to work with
• Standalone GitHub packages
• Package authorization
• Different language per package

So I think it‘s worth a shot. I will update this post as soon as possible to share the experience.

There is more

It also feels very natural to group packages, so what about creating a meta repo… Now I can handle multiple Monorepos.

github.com/mateodelnorte/meta

I need to find out what‘s most flexible and if it‘s good/bad practice to combine meta repos with Github Actions split or if one of those does everything.

Not mentioned all the monorepo versioning helper tools put there like:

Often talked about:

• bit

Proposed by PNPM:

• rushjs.io
• changesets

Update: Meta

After more evaluation I started now using meta, here a video of the inventor:

youtu.be/jvmtfSq4goo

[Fabio Gartenmann]

Tired. Tie red. I Try to visualize Kubernetes as a Monorepo for containers 🤔and what I gain by making it isomorphic …..

Usually 12 repos already available, people more productive than I‘m creative (most of times)

Will write back when getting a bit deeper into that topic, just built a Yarn module (Node CLI) and add PNPM, while normal Bash commanding and prompting was the duty). Then deploying the whole Monorepo by npx command. Let me hear from you too, I‘m interested in how you proceed then. Thank you 🙏

[Fabio Gartenmann]

also looking for best practices. In the past I was just using “subrepo”, which was fairly what I wanted, now there are monorepos which mean I don‘t need many repos but could implement a toolkit where you can grab packaged e.g. using npx (bin).

I‘m feeling (after trying Yarn that PNPM might be the wiser choice (Yarn PNP Issues and no need for package duplications optin-out to nodeLinker.

I have a package.json and a tsconfig.json for each package now, and that‘s how to separate things I guess. You just steer the whole thing via root, e.g. „pnpp build -r“ will build all the packages, or „—filter pkgName“ will address one.

So I can build my modules without having to link and npm publish and back and fro, but in the end, they are all standalone, only question:

Would you like to have 10 repos or 1 with 10 packages, which you can choose one or more from?

Abd that question is a good one and many aspects. However it‘s easy to decide again later, as changes are made cery quickly due to nice architecture of the many vendors.

Good Luck 🍀

[Kristiqn Tachev]

Usually i am doing monorepo approach only for common packages.

For example with my infrastructure

@gapi and @rxdi and @rhtml they all are monorepos

github.com/Stradivario/gapi
github.com/rxdi/rxdi-monorepo
github.com/r-html/rhtml

Then inside my application code i have separated repo for Backend and Frontend and all of the microservices that are used in this particular moment they are 10 so i got 12 Repositories

I have 1 more repository which combines all of the repositories as a git submodules

And i am using custom task management system so i can spin up all of the servers and client with single command.

If i am making a framework or a common packages i will use monorepo with lerna or yarn or latest npm

If i am making a business product i would like to have a separated github repositories which uses the packages from the monorepo

In simple words "If it is a framework or utility go with monorepo if it is a business product go with submodules"

Sayed by a guy which works also with nx monorepo using Angular and NestJS at one of my contracts. To be honest i don't like how slow it is to transpile and also the nx cache is a bit bigger so we have a big problems with CI/CD since we are caching the results from the build and in the next build we are running only affected packages.

Cheers!

[Sean Allin Newell]

I've used submodules over monorepos at two companies, I would highly recommend it. I've usually seen the root repo contain things like onboarding docs, docker compose yaml files, master Makefile, bash scripts, env var and other config etc, and then all the code is in a submodule with a separate repo.

[Fabio Gartenmann]

I agree. I’d like to point out (NodeJS dev since its invention, thank you Sir Dahl) that small reusable packages is IMO the spirit of NodeJS. It’s also fundamental in more advanced techniques like Functional Programming (e.g. Purify, SOLID principle, ….even apprenticeships in the eLearning domain use smaller reusable modules (a share among similar job descriptions, which makes it easier to switch to another).

Point of monorepo is only workspace for dev‘s, how you work not how to deliver.

from this perspective I like workspaces, makes me code comfyer without contradicting my imagination of was John invented beside of isomorphic coding, which is great, combine this with RxJs to communicate seemlessly, then make a cluster and worker.length === cpus.length and feel the not-so-single-threaded-as-smiled-at while doing way more than scrollbar-coloring-as-smiled-at-js ☺️ I‘m grateful for what we‘ve achieved so far. Open-Source will get a new framework spon. Working on it for a while now, soon ready for contributer acquiring 😏

[Fabio Gartenmann]

console:

// you probably know already ^^
$> 0.1 + 0.2

[David Armendáriz]

Yes, that sounds great! I think it is very convenient...

[Michiel Hendriks]

For strictly rooted revision control systems a mono-repo is more often a source of problems, than a solution. Most tooling cannot properly handle these kinds of mono-repos. Subversion is one of the few RCSs where any path can be a root, which solves most of the tooling issues.
What's the benefit of a mono-rope? Well, if you don't have to clean boundaries in your software components, a mono-repo is the way to go for your spaghetti.

[Pacharapol Withayasakpunt]

My monorepo is just for convenience, and does have disadvantages as you listed.

It can contain multiple languages, and I don't usually use Yarn workspace, but even then, I believe you can what to include / not include as part of workspace.

As for clean submodule, I would create an org, and transfer the repo to org; then create multiple submodules in it.