DEV Community: Pato

A new use case for git submodules

Pato — Wed, 05 Feb 2020 17:13:43 +0000

Git submodules are a gray area to me: not because of how they work, but because of identifying a situation they suit up nicely, and that is, without feeling I'm complicating things unnecessarily.

Some time ago I wrote this article in my blog, and also on dev.to which is kind of a sloppy solution to a problem that deserved a better one. So, now I've got time to revisit it, and yes: this time we approach with (oh, you got it already!) git submodules!

Now, instead of running this hard to memorize and not so extensible command I came up with before:

find . -mindepth 1 -maxdepth 1 -type d -print -exec git -C {} pull \;

Now, I can just do:

git submodules foreach command

Where command would be anything we want to run across all the repos.

So, this post is about solving what was already solved, this time with a better, more extensible approach:
A simple yet useful approach with submodules.

Git submodules

So, this is what we'll be doing:

Create a new repo
Add in every repo we want to manage as a submodule
Checkout a branch on every submodule/repo - one command!
Get the status of each submodule from top-level repo - one command!
Check all submodule repos status - one command!
Pull changes for every submodule - one command!

Alright then, let's do it!

Create a new repo

Just cd into a new dir, and init a new repo.

mkdir platform-repo && git init

Add every repo you want to control as a submodule

Repeat this as many times as repos you wish to add. Each repo will be mounted in a new directory (similarly as git clone does).

git submodule add repo_url

Checkout a branch on every submodule

Now, we need to point our submodules to a branch. Depending on your git version, by default submodules have a detached head (meaning they don't point to any branch). I'm assuming you would be pointing them to either dev or master branch, but you can point them to wichever branch you wish to.

 git submodule foreach git checkout dev

Check the status of the submodules

from the 'top-level' repo, just run:

 git status

It will show every submodule and its status. As you work on your submodules, you will see different statuses: new commits (since the moment you add them as submodules), untracked changes (not commited, nor staged), or yet pristine.

When a submodule shows new commits, it means it has them since the moment you added the submodule to the top-level repo. It works like this, because the top-level repo holds a reference to a commit in the submodule repo, and when it detects newer commits than the one it is holding its reference into, it shows the difference.
If you want it to show no new commits, you just need to create a commit in the top-level repo. You can read more about that here.

See each submodule status

To get submodules status as if it were an independant repo, you should:

 git submodule foreach git status

And you are golden.

Pull changes for every submodule

 git submodule foreach git pull --rebase --autostash origin branch_name

This way, --autostash will assure to add your changes to stash, pull, then rebase on your branch and pop changes out of stash. And all of that automatically, so you don't need to commit or stash your changes manually.

This is useful because when running commands with the git submodule foreach command expression, if the command exits with an error code, it might cancel the whole foreach chain. This way we ensure it doesn't exit as an error, by reducing chances for failling of the pull.

Other cases

For any other case, you can always cd into each submodule and see it still is a normal repository. So you can work from there as you usually do. Just keep that in mind when running git submodule foreach commands from the top level repo. I.E not all submodules might be in the same branch, etc.

Conclusions

Here is another approach to work with many repositories at once. We could reach a similar thing if working with monorepos, but this wasn't an option at the place I was working at the moment I wrote this post. So, I found this approach, which you can maybe think about 'simulating a monorepo'.

Just by adding a set of repos as submodules, you can work around it nicely.

You can read in depth about git submodules here.

This post was originally posted here

Git pull multiple repositories at once

Pato — Tue, 03 Dec 2019 17:10:26 +0000

Update: I wrote about another approach to achieve this and some extras, with git submodules. You can find it here here.

I usually work with many repos at the same time and generally group them within a same directory, named after the company.

In example, if I'm working for "Great Employer INC." I'll have a directory probably named greatemployerinc and one directory for each repo I work within. Something like:

greatemployerinc
├── greatemployerinc-web
├── greatemployerinc-auth
├── greatemployerinc-core-service
├── greatemployerinc-email-service
├── ...more repos
└── greatemployerinc-gateway

Instead of navigating each one of them with cd and then manually run git pull branch, I came out with the following command that I run from greatemployerinc -the repos containing directory.

find . -mindepth 1 -maxdepth 1 -type d -print -exec git -C {} pull \;

It will find any subdirectory in the current one (greatemployerinc), without going deeper than the first level (delimited by mindepth 1 and maxdepth 1) and execute a git pull in the corresponding git dir (if its a repo).

If you want to see more on the git -C option, you can check this post on SO

Keep in mind it will run a git pull on the current branch that each repository is actually stepping on. Also, consider the pull might fail if you have pending changes (it will be shown on the console output, anyway) and you might need to handle those cases manually, or considering adding an --autostash option to the provided command.

Wait, I can't memorize that command!

Then you and I have something in common :)

To avoid memorizing it, I created an alias. If you are working with ZSH you can just add this line to your ~/.zshrc file

alias multipull="find . -mindepth 1 -maxdepth 1 -type d -print -exec git -C {} pull \;"

Then close your current terminal and open a new one (so the changes on .zshrc take effect), go to your equivalent greatemployerinc directory and do:

multipull

And drink your favorite infusion while your repos pull from remote :)

Conclusion

So, we created a quick small command to run a pull in multiple repos at once, then we created an alias for it so it is more easy to run later.

If you are willing to take it further, you can tweak it a little bit around if you want to customize the branch you pull from/into, run a different command, or try and make it run in parallel.

Updates

If you are interested on "resourcing" your terminal without closing/opening it, see this comment from JPBlancoDB

If you are interested on more complex scenarios, you can use a tool like myrepos just like Caleb Maclennan points out in this comment

Recording a terminal session

Pato — Thu, 28 Nov 2019 19:17:27 +0000

Whether you are writing a tutorial post on some blog, willing being able to reproduce a set of commands, just taking personal notes, or whatever the reason, it is helpful to record your terminal session, with all its commands and outputs.

For such a task, I've come across the script command, which according to its man page, just makes a typescript of terminal session.

You can start with a minimal expression, such as

script pepe

Where pepe will be the output file.

Once you are done recording your terminal work, you can stop recording with exit.

Then you can cat pepe and see how it went.

You might find unwanted text and keystrokes as Script started, output file is pepe. I recommend seeing the man pages (man script) to learn how to improve the expected result and adapt it to your needs and taste.

In my case, I'm writing complementary material for an online course I'm working on, so it is helpful to record the terminal, instead of copying and pasting as I go, so I can quickly create guides. The form of the command I'm more frequently using is based on:

script -q -k pepe

The -q option suppress messages such as the script started and the script finished.

The -k option logs keys sent to the program as well as output, omitting the datetime stamp.

Super interesting thing: you can record sessions with a particular format and reproduce them with scriptreplay. Still thinking of ways this can help me create course material.