It's a common scenario we lost our local projects because of problems in our machine. Our Laptop/Desktop can break down and we lost a lot of data. It's a painful situation, recovery not only all data, photos, software's, etc. But, recovery all our repositories too.
To our happiness, all repositories are storage/save in cloud through GitHub if we use Github as hosting service.
However, if we work simultaneously on various projects could be a little boring clone all repositories one-by-one.
Automate tasks is a great way to make a better use of our time. And shell script is a good resource to achieve this type of automation.
In this article we will create a shell script that automatically clone all repositories of your GitHub account.
Shell Script
Shell script is a script language used in command line environment. It allow us automate terminal commands in Unix based systems like Linux and MacOS.
Let's start
This article assumes that you have a basic knowledge about terminal commands, so is not the focus here explain what instructions like touch, mkdir, etc do.
First of all, we need to create a new script file.
touch clone-github-repositories.sh
Note that the ".sh" is the extension of shell script files, without it the interpreter cannot understand that this file is a shell script.
After, we need open this file in a text editor of your preference (vi, nano, etc).
nano clone-github-repositories.sh
In the editor we need write in the first line something very important to our script: indicate the interpreter of the script. In my case, I will use bash, but, exists others like sh, ksh, etc.
So, the first line of our script will be the following:
#!/bin/bash
We will create a directory where we'll clone all our repositories. I will call this directory "projects", but you can name it like you want.
So the next line of our script will be the this command:
mkdir projects
Like I said, we will clone our repositories inside projects directory, so we need enter to the projects directory, the following line of script will be:
cd projects
Okay, so until now we have some simple instructions in our script, we need have access to all repositories from our GitHub account, we could go to our GitHub profile and copy the URL repositories one-by-one, but, this is not an agile way, right? Here, enter in scene the GitHub REST API.
The GitHub REST API allow us make integrations, automation and much more.
Our next line in script will be a request to GitHub API using curl:
curl -s https://api.github.com/users/{your-github-username}/repos
Replace {your-github-username} by your GitHub username.
If you run this command solo in terminal the return will be a JSON content with a array of objects containing all properties of your repositories. Below an example of the JSON that return from the above command.
Between the many properties we have two especially important: ssh_url and clone_url.
This properties contains the URL's that we can use to clone the repositories. If you use ssh feature to clone your projects, you can use the "ssh_url", else, the "clone_url" will work the same way. In my case, I use SSH to clone my repositories.
Now, we have to work with some resources to manipulate content of files, because we don't need use all information that return from GitHub API.
Here, "grep" is our best friend. grep will search inside a file by a string/pattern that we inform.
We will refactor our script line to the following:
curl -s https://api.github.com/users/{your-github-username}/repos | grep ssh_url
Remember that if you don't use SSH you must change the "ssh_url" by "clone_url".
The output of this instruction will be something like this (if you run this directly in terminal):
To clone a project we need only the URL, so we need filter this grep return.
We will use the command "cut" with a delimiter to remove the unnecessary sections from the lines.
Refactoring our instruction, now the command will be this:
curl -s https://api.github.com/users/{your-github-username}/repos | grep ssh_url | cut -d'"' -f 4
-d is a flag for field delimiter and we use the double quotes (") as delimiter.
-f is a flag for fields that select only the field (or range of fields) informed. And 4 is the specified index of field line that we want extract.
If you run this instruction in terminal the output is like this:
Now, we have the exact URL to clone our repositories.
So, we will refactor the line to save the output list in a variable.
repos=$(curl -s https://api.github.com/users/{your-github-username}/repos | grep ssh_url | cut -d'"' -f 4)
repos is the name of variable and the "$" symbol indicate to the shell that this is a variable, we have a list of URL's in this variable.
We will need apply some programming logic to do the task, iterable the repos list and for each repository of the list we will clone the repository.
for repo in $repos
do
git clone $repo
done
The full script is below:
#!/bin/bash
mkdir projects
cd projects
repos=$(curl -s https://api.github.com/users/gniches/repos | grep ssh_url | cut -d'"' -f 4)
for repo in $repos
do
git clone $repo
done
It's done. Now we need run the script to test if our script is working as expected.
bash clone-github-repositories.sh
If everything is ok, now you have a directory called projects with all your repositories inside.
Concluding
In this article we learned how to create a shell script to clone all our repositories from GitHub account using GitHub API.
Shell script is particularly useful to automate process and tasks, like monitoring servers, create periodic backups of databases, send e-mails and much more.
Reviews, suggestions and/or contributions to this article are encouraged and welcome.
Top comments (3)
Bem legal, mas me permita apenas uma pequena contribuição. Podes fazer isso com um único comando, para o qual podes criar um alias. Eu uso como alias: meusrepos
Quando rodo isso no bash ele clona todos meus repositórios, usando minha chave para SSH, com o comando abaixo:
curl "https://api.github.com/users/<minha_conta>/repos" |grep -e 'ssh_url*' |
cut -d \" -f 4 |
xargs -L1 git clone
Separei em mais linhas, nos pipes, para facilitar a leitura.
Sensacional, Carlos! Admito que nem passou pela minha cabeça utilizar um alias para chegar nesse resultado. Obrigado pela contribuição.
Outro detalhe é que esse comando só clona os repositórios públicos, para obter os privados precisa do Github client. Uso outro alias para os privados, com o comando:
gh repos | grep -e "/" | cut -d / -f 2 | cut -d " " -f 1 | xargs -L1 gh repo clone