DEV Community

Fernando Paladini
Fernando Paladini

Posted on

Back Up GitHub Repositories to GitLab with Python

#python #devops #tutorial #gitlab

Keeping every repository on one hosting platform creates a single point of failure for your source code. A second copy helps, but a backup script that treats every repository as public can create a much worse problem: accidental exposure.

This tutorial walks through backup-github-to-gitlab, an open-source Python tool that lists a GitHub user's repositories, creates matching GitLab projects, and mirrors branches and tags over SSH. The workflow defaults private when it cannot determine visibility, supports a dry run, and can be run again to push incremental updates.

TL;DR

The smallest useful workflow is:

  1. Install Python dependencies and Git.
  2. Create a YAML configuration and put API tokens in environment variables.
  3. Run python backup.py --dry-run to inspect the plan.
  4. Run python backup.py after checking the proposed visibility.

The tool is a repository backup, not a complete disaster-recovery system. It does not transfer Git LFS objects, organization repositories, or wikis in its current version.

Prerequisites

You need Python 3.11 or newer, Git on your PATH, and SSH keys registered with both GitHub and GitLab. The project also expects a GitHub token with repo scope and a GitLab token with api scope. Those scopes are used by the API clients to list repositories and create GitLab projects.

The repository is MIT-licensed and the current source is available in backup-github-to-gitlab on GitHub. The dependency list includes PyGithub, python-gitlab, python-dotenv, Rich, and PyYAML.

Install the tool

Clone the project and install its dependencies:

git clone git@github.com:paladini/backup-github-to-gitlab.git
cd backup-github-to-gitlab
python -m pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

The script exposes a normal command-line help screen. python backup.py --help shows options for dry runs, repository glob filters, verbose Git output, forks, archived repositories, and an alternate configuration path.

Configure credentials and usernames

Copy the checked-in configuration example to config.yaml and set the GitHub and GitLab usernames:

github:
  username: your-github-username

gitlab:
  username: your-gitlab-username
  url: https://gitlab.com

backup:
  include_forks: false
  include_archived: true
  temp_dir: ./tmp
Enter fullscreen mode Exit fullscreen mode

Put the tokens in your process environment or in a local .env file that is not committed:

GITHUB_TOKEN=your-github-token
GITLAB_TOKEN=your-gitlab-token
Enter fullscreen mode Exit fullscreen mode

The application loads .env at runtime. The tokens are not read from config.yaml, and the README states that they are not written to logs. Treat any token copied into a shell history, terminal recording, or issue as compromised and rotate it.

Preview the backup safely

Before making a GitLab project or pushing objects, run the dry-run mode:

python backup.py --dry-run
Enter fullscreen mode Exit fullscreen mode

The runner still queries GitHub and checks whether each matching GitLab project exists, but it returns a dry_run result instead of creating projects, cloning repositories, or pushing Git data. A typical report distinguishes an existing project from a project that would be created, and shows whether the project would be public or private.

For a narrower preview, use a glob filter:

python backup.py --dry-run --filter "client-*"
Enter fullscreen mode Exit fullscreen mode

This is useful for a first run against a small set of repositories. The default configuration skips forks and includes archived repositories. The command-line flags can override those choices when you intentionally want a different scope.

Run the mirror

After reviewing the dry-run output, execute the backup:

python backup.py
Enter fullscreen mode Exit fullscreen mode

For each repository, the runner follows this sequence:

  1. Ask GitHub for repositories owned by the configured user.
  2. Check whether the matching GitLab project exists.
  3. Create the GitLab project with the visibility reported by GitHub, or reuse its SSH URL.
  4. Run git clone --mirror from the GitHub SSH URL.
  5. Run git push --mirror from the temporary mirror to GitLab.
  6. Remove the temporary clone, including read-only Git files on Windows.

The mirror flags matter. A normal working-tree clone is optimized for editing one branch. A mirror clone contains all refs, and a mirror push updates the destination refs to match the source. That makes the operation appropriate for a source backup, while also meaning that the destination should be dedicated to this mirror. Manual changes made directly in GitLab can be overwritten by a later run.

Why reruns are safe enough for routine backups

The tool checks for an existing project before creating one. On a later run it gets that project's SSH URL, creates a fresh temporary mirror, and pushes the current GitHub refs. This makes the process idempotent at the project level: rerunning it does not create duplicate GitLab projects.

It is not a versioned backup in the archival sense. If a branch or tag is removed from GitHub, git push --mirror can remove the corresponding ref in GitLab. If you need point-in-time recovery, add a separate retention policy such as dated bare-mirror snapshots or storage-level backups.

Visibility and security boundaries

Visibility is the most important safety property in this workflow. The GitHub client maps the API's private value into a repository record. The GitLab client creates a project as private when that value is true and as public otherwise. The README also documents a private-by-default policy when visibility cannot be determined, so verify the actual project settings before pushing sensitive code.

The tool uses SSH for Git transport and API tokens for metadata operations. That separates repository content transfer from API authentication, but it does not remove the need to protect either credential. Use a dedicated GitLab account or group when appropriate, review token scopes, and test the process with a non-sensitive repository first.

The dry run is a planning aid, not a permission boundary. It can show what the program intends to do, but it cannot prove that a token has the right permissions or that an SSH key can push. A small real run is the discriminating test.

Failure modes to expect

The configuration file is missing

The default path is config.yaml. Create it from config.example.yaml, or pass another file with --config. The validator reports missing usernames and missing GITHUB_TOKEN or GITLAB_TOKEN before the runner starts.

Git authentication fails

The API tokens do not authenticate the Git SSH connection. Confirm that ssh -T git@github.com and the equivalent GitLab SSH check work for the account that owns the destination projects. With --verbose, the tool exposes Git command output that can help isolate an SSH problem.

A GitLab project already contains unrelated data

The project intentionally pushes a full mirror. Do not point it at a destination that people edit independently. Create a dedicated project or confirm that replacing its refs is acceptable.

Rate limits or transient API errors occur

The GitHub client waits for a core API rate-limit reset. GitLab HTTP 429 responses are retried up to four times with a 60-second delay. A failure after those retries is reported for that repository, while the runner continues processing the list.

Limitations

The current README explicitly lists three limits:

  • Git LFS objects are not transferred by git clone --mirror.
  • Organization repositories are not included in version 1.
  • GitLab pull mirroring for private repositories requires GitLab Premium, and the planned GitHub Actions alternative is not implemented here.

Wikis are also listed as a future roadmap item. If any of these are part of your recovery objective, treat this tool as one layer of the backup plan and verify those assets separately.

FAQ

Does the tool copy private repositories as public?

The implementation maps GitHub visibility to the GitLab project. Private repositories are created as private. Still inspect the dry-run output and destination settings before using it for sensitive repositories.

Can I back up only one repository?

Yes. Use a matching glob, such as --filter "my-project", and run a dry run first.

Does it need a GitHub Actions workflow?

No. It runs locally and uses the GitHub and GitLab APIs plus SSH Git operations.

Is this a replacement for snapshots?

No. It keeps a GitLab mirror current. Add retention and independent storage if you need historical recovery after deletions or corruption.

Takeaway

backup-github-to-gitlab is a focused way to create a second Git hosting copy without writing custom API and Git orchestration code. Its strongest design choices are the dry-run mode, visibility-aware project creation, mirror refs, and cleanup of temporary clones. Start with one harmless repository, verify both API and SSH authentication, inspect the proposed visibility, and only then expand the filter.

What would you add first for your own backup policy: Git LFS support, organization repositories, or retention-aware snapshots?

Disclosure: AI assistance was used to organize this tutorial and review its wording. The commands, behavior, limitations, and project details were checked against the repository's current public README, source files, configuration example, dependency list, license, and public GitHub metadata.

Top comments (0)