TIL different ways to clone multiple git repos at once

#todayilearned #github #gitlab #linux

As someone who frequently works with GitHub and GitLab repositories, I often need to get an overview of groups. At my workplace, we use GitLab’s group- and subgroup function to organize our team-specific and customer-specific structures.

Recently, I needed to clone all 27 repositories in our microservice-project. Doing this manually would require me to copy the URL for each repository and run “git clone \$URL” on the command line, which is time-consuming and tedious.

Fortunately, I discovered a tool called ghorg that simplifies this process. With just one command, I was able to clone entire GitHub organizations or GitLab (sub)groups. Here’s the command I used:

ghorg clone customergroup/services --base-url=https://git.example.com --scm=gitlab --token=$TOKEN --preserve-dir -p .

This command clones all repositories in the group customergroup/services, connecting to our private GitLab instance, using the specified token $TOKEN to authenticate. The --preserve-dir option clones the repos in a directory structure that matches GitLab groups, so repository foo will be cloned into customergroup/services/foo.

ghorg clone customergroup/services --base-url=https://git.example.com --scm=gitlab --token=$TOKEN --preserve-dir -p .

 +-+-+-+-+ +-+-+ +-+-+-+-+-+
 |T|I|M|E| |T|O| |G|H|O|R|G|
 +-+-+-+-+ +-+-+ +-+-+-+-+-+

*************************************
* SCM : gitlab
* Type : org
* Protocol : https
* Location : ./
* Concurrency : 25
* Base URL : https://git.example.com
* Preserve Dir : true
* Config Used : none
* Ghorg version : v1.9.3
*************************************
27 repos found in customergroup/services

Success cloning repo: https://git.example.com/customergroup/services/foo.git -&gt; branch: main
Success cloning repo: https://git.example.com/customergroup/services/foo-audit-report.git -&gt; branch: master
Success cloning repo: https://git.example.com/customergroup/services/foo-tools.git -&gt; branch: main
Success cloning repo: https://git.example.com/customergroup/services/foo-base.git -&gt; branch: main
Success cloning repo: https://git.example.com/customergroup/services/foo-pages.git -&gt; branch: main
Success cloning repo: https://git.example.com/customergroup/services/foo-helm.git -&gt; branch: master
Success cloning repo: https://git.example.com/customergroup/services/foo-application-base.git -&gt; branch: master

But I’m also a great fan of the gitlab-cli and it has a similar feature, too!

Running the following command does the same as ghorg does:

GITLAB_HOST=git.example.com glab repo clone --group customergroup/services -p

The syntax is slightly different. With --group $GROUP you define the group you want to clone; with -p you preserve the group-structure.

&gt; GITLAB_HOST=git.example.com glab repo clone --group customergroup/services -p
Cloning into 'customergroup/services/foo'...
remote: Enumerating objects: 59, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 59 (delta 0), reused 0 (delta 0), pack-reused 54
Receiving objects: 100% (59/59), 13.36 KiB | 1.03 MiB/s, done.

Both tools produce the same folder structure with the cloned repositories:

&gt; tree -d -L 3
.
└── customergroup
    └── services
        ├── foo
        ├── foo-audit-report
        ├── foo-tools
        ├── foo-base
        ├── foo-pages
        ├── foo-helm
        └── foo-application-base

There are many more tools that do this, like myrepos, vcsh or gitlabber, but I didn’t try these.