DEV Community

Cover image for Automatic Backup and Synchronization of All Repositories from a Git Account or Organization
Ivaj O'Franc
Ivaj O'Franc

Posted on • Edited on • Originally published at dev.to

Automatic Backup and Synchronization of All Repositories from a Git Account or Organization

#git #github #bash #automation 
id: 2763189
title: "Automatic Backup and Synchronization of All Repositories from a Git Account or Organization"
published: true
tags: ["git", "github", "bash", "automation"]
series: I Fixed It and I Don't Know How
description: "Automatic Backup and Synchronization of All Repositories from a Git Account or..."
canonical_url: "https://dev.to/ivajofranc/automatic-backup-and-synchronization-of-all-repositories-from-a-git-account-or-organization-2ehi"
cover_image: "https://media2.dev.to/dynamic/image/width=1000,height=420,fit=cover,gravity=auto,format=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F522asln7hhm2jpp96qar.png"
Enter fullscreen mode Exit fullscreen mode

Automatic Backup and Synchronization of All Repositories from a Git Account or Organization

🇪🇸 Read this post in Spanish

In this tutorial, I’ll explain how I cloned and kept all repositories from a GitHub account/organization synchronized. It also works for GitLab, Bitbucket, or any Git server that exposes an API and supports git clone and git pull.

🎯 Goal

  • Download all repositories (public and private) from an account/organization.
  • Store them in a local folder (optionally synced with OneDrive, Dropbox, etc.).
  • Update them with a single command (git pull in each one).
  • Log everything into a file for auditing purposes.

🛠️ Environment Used

  • OS: Windows 10/11
  • Terminal: Git Bash (included with Git for Windows)
  • Required tools:

🔑 Steps Taken and Lessons Learned

1️⃣ Creating the Access Token

  • Create a Personal Access Token (PAT) in GitHub with permissions:
    • repo (for private repos)
    • read:org (for organizations)

Problem: Using the organization endpoint on a user account returned 404 Not Found.

Solution: Differentiate between user (/users/username/repos) and organization (/orgs/org/repos).

2️⃣ Initial Cloning Script 

ORG_NAME="Example-Organization"
OUTPUT_DIR="."
REPOS_FILE="$OUTPUT_DIR/repos.txt"
GITHUB_TOKEN="YOUR_TOKEN_HERE"

mkdir -p "$OUTPUT_DIR"
cd "$OUTPUT_DIR" || exit 1

curl -s -H "Authorization: token $GITHUB_TOKEN" "https://api.github.com/orgs/$ORG_NAME/repos?per_page=100&type=all" | jq -r '.[].clone_url' > "$REPOS_FILE"

while read -r repo_url; do
    git clone "$repo_url"
done < "$REPOS_FILE"
Enter fullscreen mode Exit fullscreen mode

✅ Works for cloning all repos.

❌ Fails on repeated execution: fatal: destination path already exists.

3️⃣ Improved for Synchronization (git pull) 

repo_name=$(basename "$repo_url" .git)

if [ -d "$repo_name/.git" ]; then
    git -C "$repo_name" pull
else
    git clone "$repo_url"
fi
Enter fullscreen mode Exit fullscreen mode

✅ Updates if it already exists.

❌ Fails if folder name ≠ repo name (case sensitivity issues).

4️⃣ Robust Solution 

if [ -d "$repo_name/.git" ]; then
    (cd "$repo_name" && git pull)
elif [ -d "$repo_name" ]; then
    echo "⚠️ Folder $repo_name exists but is not a Git repo"
else
    git clone "$repo_url"
fi
Enter fullscreen mode Exit fullscreen mode

✅ Correctly updates all existing repos.

5️⃣ Logging 

LOG_FILE="$(pwd)/sync-repos.log"

echo "$(date) - Starting synchronization" | tee -a "$LOG_FILE"

while read -r repo_url; do
    repo_name=$(basename "$repo_url" .git)
    if [ -d "$repo_name/.git" ]; then
        echo "📥 Updating $repo_name" | tee -a "$LOG_FILE"
        (cd "$repo_name" && git pull >> "$LOG_FILE" 2>&1)
    else
        echo "📦 Cloning $repo_name" | tee -a "$LOG_FILE"
        git clone "$repo_url" >> "$LOG_FILE" 2>&1
    fi
done < "$REPOS_FILE"

echo "$(date) - Finished" | tee -a "$LOG_FILE"
Enter fullscreen mode Exit fullscreen mode

📂 Backup Location

  • OUTPUT_DIR="." → saves in the current folder.
  • Fixed location in Windows + Git Bash: 
  OUTPUT_DIR=/c/Users/username/OneDrive/Backups/github
Enter fullscreen mode Exit fullscreen mode

🔄 Adapting for GitLab or Bitbucket

  • GitLab: https://gitlab.com/api/v4/groups/<group>/projects
  • Bitbucket: https://api.bitbucket.org/2.0/repositories/<user>
  • Adjust authentication (Bearer token, Basic Auth, etc.).

💡 Final Recommendations

  • Test first in a sandbox folder.
  • Avoid spaces in paths or escape them with \.
  • Handle network errors with set -e or validations.
  • Schedule execution with Task Scheduler (Windows) or cron (Linux).
  • Keep logs for historical records.

📌 Useful Links

Top comments (0)