A new use case for git submodules

Git submodules are a gray area to me: not because of how they work, but because of identifying a situation they suit up nicely, and that is, without feeling I'm complicating things unnecessarily.

Some time ago I wrote this article in my blog, and also on dev.to which is kind of a sloppy solution to a problem that deserved a better one. So, now I've got time to revisit it, and yes: this time we approach with (oh, you got it already!) git submodules!

Now, instead of running this hard to memorize and not so extensible command I came up with before:

find . -mindepth 1 -maxdepth 1 -type d -print -exec git -C {} pull \;

Now, I can just do:

git submodules foreach command 

Where command would be anything we want to run across all the repos.

So, this post is about solving what was already solved, this time with a better, more extensible approach:
A simple yet useful approach with submodules.

Git submodules

So, this is what we'll be doing:

1. Create a new repo
2. Add in every repo we want to manage as a submodule
3. Checkout a branch on every submodule/repo - one command!
4. Get the status of each submodule from top-level repo - one command!
5. Check all submodule repos status - one command!
6. Pull changes for every submodule - one command!

Alright then, let's do it!

Create a new repo

Just cd into a new dir, and init a new repo.

mkdir platform-repo && git init

Add every repo you want to control as a submodule

Repeat this as many times as repos you wish to add. Each repo will be mounted in a new directory (similarly as git clone does).

git submodule add repo_url

Checkout a branch on every submodule

Now, we need to point our submodules to a branch. Depending on your git version, by default submodules have a detached head (meaning they don't point to any branch). I'm assuming you would be pointing them to either dev or master branch, but you can point them to wichever branch you wish to.

 git submodule foreach git checkout dev

Check the status of the submodules

from the 'top-level' repo, just run:

 git status

It will show every submodule and its status. As you work on your submodules, you will see different statuses: new commits (since the moment you add them as submodules), untracked changes (not commited, nor staged), or yet pristine.

When a submodule shows new commits, it means it has them since the moment you added the submodule to the top-level repo. It works like this, because the top-level repo holds a reference to a commit in the submodule repo, and when it detects newer commits than the one it is holding its reference into, it shows the difference.
If you want it to show no new commits, you just need to create a commit in the top-level repo. You can read more about that here.

See each submodule status

To get submodules status as if it were an independant repo, you should:

 git submodule foreach git status

And you are golden.

Pull changes for every submodule

 git submodule foreach git pull --rebase --autostash origin branch_name

This way, --autostash will assure to add your changes to stash, pull, then rebase on your branch and pop changes out of stash. And all of that automatically, so you don't need to commit or stash your changes manually.

This is useful because when running commands with the git submodule foreach command expression, if the command exits with an error code, it might cancel the whole foreach chain. This way we ensure it doesn't exit as an error, by reducing chances for failling of the pull.

Other cases

For any other case, you can always cd into each submodule and see it still is a normal repository. So you can work from there as you usually do. Just keep that in mind when running git submodule foreach commands from the top level repo. I.E not all submodules might be in the same branch, etc.

Conclusions

Here is another approach to work with many repositories at once. We could reach a similar thing if working with monorepos, but this wasn't an option at the place I was working at the moment I wrote this post. So, I found this approach, which you can maybe think about 'simulating a monorepo'.

Just by adding a set of repos as submodules, you can work around it nicely.

---

You can read in depth about git submodules here.

This post was originally posted here

Comments

[will-m-rose]

Not sure if it's because I'm on Git for Windows, but a couple times above you list commands such as:

git foreach submodule git status

When I run that my shell says:

git: 'foreach' is not a git command.

I just had to swap foreach and submodule to make it work.

So this works:

git submodule foreach git status

[Pato]

You are right, thank you for pointing this out!

I miss spelled the command. git submodule foreach git status is the actual command. I'll update the post.

[neoan]

For people using PHPStorm: by now the IDE handles submodules really nice. If the project allows, I treat every component as it's individual repo. This fundamentally changed my approach for reusability, testing (especially CI) and even deployment. There would be a lot to say about strategy, but what I like about this article is that you grasp the general idea and can then freely think of how you would apply that to your versioning strategy.

[Pato]

Hi neoan! Thanks for commenting :)

In my particular case, i'm working with many projects in different languages, so i'm not using a single IDE. Actually, i'm more likely to do changes from the console (vim + some plugins) than opening IDEs since i work with many projects at the same time and it consumes to much ram.

Anyway, PHPStorm is a great IDE.

Hopefully this might give a little bit more knowledge about git submodules and things you can do with them.

[Dileepa Mabulage]

Hi, can PRs be created between the sub-modules and the main module? I am looking for the need to maintain two separate repositories, parent and child. The parent has all the features, but the child does not. So I need to make PR from parent to child for some features. Is this a possible thing in GitHub?

[Shaun]

Clever idea I like it.

[Pato]

Thank you! I'm glad you like it :)