tsrc — Handling multiple git repositories without losing your HEAD

#git #showdev #enterprise

Git has become the mainstream source code manager and is the de facto standard for many projects, both open and closed source.

However, it does not scale well with huge code bases commonly found in the enterprise world.

Some companies decide to put everything in one giant repository. By doing so, they often end up patching existing source control software as Facebook did with Mercurial.

Another way to do things is to keep using good old git: by splitting the sources across multiple repositories.

Managing complexity

In the "multi-repo" case, how do you keep track of all the repositories, and how do you synchronize changes among them?

One popular method is to elect a 'master' repository and manage the others ones through the use of git submodules. Unfortunately, it is not always possible to implement this master/slaves design. Furthermore, the ergonomics of git submodule are not always on par.

tsrc

Enter tsrc. We use it every day at Tanker to manage our many sources.

At its core lies the concept of a manifest. It's a simple YAML file that lists repositories by their relative paths in the workspace (src) and by URL:

repos:
  - src: foo
    url: git@gitlab.local:acme/foo

  - src: bar
    url: git@gitlab.local:acme/bar

That means all repositories are listed in one place.

If you have multiple teams, you may want to use groups so that each team can select which subset of the repositories they need. Each group has a name and a list of repositories:


groups:
  first_group: 
    repos: [a, b, c]
  second_group:
    repos: [d, e]

This approach is known to work with at least 300 different git repositories :)

Usage

To use tsrc, all you have to do is put the manifest itself in a git repository, and then use the following
commands to create a new workspace:

$ mkdir ~/work
$ cd work
$ tsrc init git@gitlab.local:acme/manifest.git

You can select one or several groups with the -g/--group option.

Making sure all the repositories are up to date

Once the workspace is created, you can update all the repositories at once by using tsrc sync.

The manifest itself is updated first.
If a new repository has been added to the manifest, it is cloned.
Lastly, the other repositories are updated.

Note that tsrc sync only updates the repositories if the changes are trivial. tsrc will not touch any repository if:

The repository is dirty
The local branch has diverged from upstream
There is no remote tracking branch

It is up to you to solve these issues any way you see fit. This way, there is no risk of data loss or sudden conflicts to appear. tsrc focuses on doing simple tasks right and never tries to do smart things.

Lastly, so that you know where manual intervention is required, tsrc sync also
displays a summary of errors at the end:

$ tsrc sync
...
* (2/2) spam/eggs
=> Fetching origin
=> Updating branch
fatal: Not possible to fast-forward, aborting.
Error: Failed to synchronize workspace
* spam/eggs: updating branch failed

Managing merge requests

As a bonus, tsrc is also able to automate review workflows, whether you are using GitHub or GitLab.

More info about all of this in the documentation.

Conclusion

We hope you'll give tsrc a try and that you'll find it a handy tool for your own projects.

tsrc development happens on GitHub.

Feel free to contribute, open issues, and/or give us feedback in the comment section.

PS: tsrc is made with love by the same talented people working at tanker.io. We provide an open-source privacy solution that's as user-friendly as possible. Sign up and take it for a spin here: https://tanker.io/.