DEV Community: g.bak

GitHub Restore and Disaster Recovery - Better Get Ready

g.bak — Mon, 03 Apr 2023 16:20:52 +0000

First, in case you missed it, you may want to read my previous blog post about GitHub backup. I described the potential threats to GitHub environments: human errors (mistakes or malicious actions), outages, cyber attacks… and I also provided a helpful GitHub Backup Cheat Sheet:

The Ultimate Guide to GitHub Backups

While backup is an essential part of any disaster recovery strategy, the organization should pay special attention to some important aspects. In this blog post, I will cover all the aspects concerning building a reliable and agile Disaster Recovery (DR) strategy that will keep your business continuity with ease, and we will also see 3 use cases.

Without further ado, let’s review what you should include in your GitHub disaster recovery strategy.

What Should your GitHub Disaster Recovery Strategy Include?

Sensitive data identification

A good disaster recovery plan should identify what data requires stronger security measures, how often that data should be backed up, and who is authorized to access the original copy and its backups.

It means that the organization should identify its active and important GitHub projects, taking into account metadata as well.

Recovery Point Objective (RPO) and Recovery Time Objective (RTO)

RPO and RTO are two key metrics when it comes to building a recovery strategy: they determine how much data the company can afford to lose and how quickly it should restore systems and processes in the event of failure.

Recovery Point Objective

RPO determines the time between backups and the amount of data the company can lose between those backups. So it’s critical to assess the target frequency of backup. For example, if the company can tolerate 8 hours of data loss (a normal working day) in a disaster, its RPO is 8 hours. In this case, the company can back up its data once a day.

On the other hand, if the company’s RPO is 5 minutes, the company must back up its data much more frequently.

Recovery Time Objective

RTO is another critical metric, which refers to the maximum amount of time that a business process can be down after a disaster strikes. So, in simple language, it’s the set target time for the recovery of IT and business operations. For example, if the organization sets its RTO to 5 hours, it means that the organization has to restore all its critical data within those 5 hours after a disaster. During those 5 hours, the organization will need to identify and mitigate the cause of downtime and get back to the normal working process within those 5 hours.

RTO helps to find the balance between cost efficiency and the preparation process in the event of failure. If your RTO is low, it means that you’re highly prepared for a disaster and have a comprehensive Disaster Recovery strategy permitting you to recover the data fast and guaranteeing business continuity. A higher RTO, on the other hand, may appear less expensive to maintain, but when a disaster strikes, your business may finish investing a lot more time and money in data recovery.

To find this tradeoff, it is assumed that high-priority and crucial services need a lower RTO, so they’re restored in the first place, and should be backed up with more frequency and replicated between different storages.

How to Check RTO and RPO?

It takes a rigorous examination of all your systems to determine an RTO. To be effective, your RTO must be realistic. You can't just pick a number and put it in your SLA - it should be measured and checked on a regular basis.

To do so, you should review your backup configuration and Disaster Recovery plans regularly. A regular check should include the assessment of backup and recovery processes and the review of backup logs and reports: what’s the backup frequency? Do you use replication? Incremental backups? Granular recovery for instant access to your critical data?

This analysis will permit you to make improvements to your RTO and RPO metrics on time.

At the same time, you shouldn’t forget about regular testing and simulations to ensure that your backup configuration is effective and meets the RTO and RPO objectives you set.

Distribution of Staff Roles

The company’s plan should include the names, contact details, and obligations of team members who are responsible for disaster recovery processes. So there should be people (and they should be instructed thoroughly) who are responsible for declaring a disaster, reporting to management, dealing with customers, third-party vendors, the press, and, of course, those who are in charge of managing the crisis and recovering from it.

Disaster Recovery Procedures

It is crucial for the company to establish and document the set of actions and plans aimed at reducing and minimizing the impact of the catastrophic event. It’s worth remembering that the first few hours are the most critical, so the team must know for sure what to do to restore normal operations as quickly and efficiently as possible.

Thus, CTOs and security leaders should outline all the steps that are crucial to take in the event of a disaster, including data backup and recovery, emergency communication protocols, and activation of contingency plans.

Communication Plans for the Events of Failure

In case of a disaster, a company needs to have a comprehensive communication plan for delivering the information to stakeholders and other affected parties, including management, employees, media, customers, and compliance authorities. In an ideal world, this plan should include the most appropriate communication channels, message templates, and a person or a team who will be responsible for coordinating and disseminating information during the event of failure. This is the best way to keep stakeholders' and customers’ confidence and loyalty.

Moreover, it’s worth continuously evaluating and improving the company’s communication plan according to the feedback from stakeholders and experience from previous disaster events (if they took place).

Must-Have Features for Your GitHub Restore Plan

Now, let's take a look at complete Disaster Recovery options and some GitHub restore features important not just to recover in case of a failure, but also for everyday work.

Any-to-any GitHub restore - choose your destination

Regardless of whether you use the on-premise or SaaS GitHub version, you should be able to freely restore your data to any instance, from any instance: from self-hosted to cloud and conversely. You should also have the possibility to restore your GitHub data to your local machine or crossover to another git hosting service provider (like Bitbucket or GitLab). To this end, you should be able to restore all repositories in bulk and choose the exact point in time for the copy you want to restore. Thus, you need to have long-term retention. GitHub SLA limits recovery to only up to 90 days after repositories have been deleted.

Granular recovery - for daily operations

Quite often companies protect their GitHub Enterprise Server by backing up the entire underlying VM. In this case, the company would have to recover the entire virtual machine in case of a failure, which can take a lot of time: all the GitHub data must be recovered before being accessible again. Thus, to bring speed, efficiency, and peace of mind to daily operations, it’s important to have the possibility to recover only chosen, individual GitHub repositories and metadata instead. This is what is called granular recovery, and it is a powerful tool against developer errors to ensure quick and immediate recovery of key data and uninterrupted workflow.

GitHub Disaster Recovery Use Cases

When you're analyzing third-party backup tools to protect your GitHub repositories and metadata, ensure that the backup provider you choose can guarantee data recoverability and accessibility under any disaster scenario. Moreover, you should get step-by-step guidance on what recovery option you can use for faster data restoration in any failure event. Let's look at the possible scenarios your GitHub environment can face and how to prevent data loss and assure your DevOps team's workflow continuity.

Use Case # 1 - GitHub Outage

We have already mentioned that, as with any other git service provider, GitHub can experience an outage. Do you want to stop your working process due to a long-term downtime of the GitHub cloud infrastructure? Surely, not. Then you need to have a few options to overcome that outage:

Restore your GitHub repositories as a file to your local machine. You can perform that operation from the last copy or just from some appropriate point in time.
Recover all your critical GitHub data to GitHub on-premise instance. For that reason, you can use the same or a new instance.
Use cross-over recovery. Migrate and restore your GitHub repositories and metadata to other git hosting services, like Bitbucket or GitLab.

Use Case # 2 - Your Infrastructure Outage

Ensure that your backup solution is a multi-storage system and allows you to assign many cloud and on-premise data stores. In this case, you can follow the 3-2-1 backup rule and restore GitHub data from a copy once one of your servers is down.

Use Case # 3 - Your Backup Provider's Infrastructure Outage

Every backup provider lives in security. That's why it is obvious that it should have a Disaster Recovery response to the scenario if its infrastructure goes down. Ensure your backup vendor has a Disaster Recovery plan of its own infrastructure.

And One More Thing...

GitHub disaster recovery and data restoration depend on how well-prepared your GitHub backup is for any potential failure.

GitProtect.io, a DevOps backup and Disaster recovery software for GitHub, GitLab, Bitbucket, and Jira, provides all the above-mentioned features and instant Disaster Recovery.

And get 95% OFF after the 14-day free trial with the promo code: BACKUPDAY95 (valid until June 1st, 2023)

If you want to find out how to create a backup plan for a reliable Disaster Recovery plan, remember to check out GitHub Backup Best Practices.

The Ultimate Guide to GitHub Backups

g.bak — Mon, 19 Dec 2022 13:32:37 +0000

In such a fast-developing world, it becomes more and more important to make sure the source code and its metadata are backed up in case of an emergency. Learn everything you need to know about how to backup a GitHub repository.

Learn more about why backup GitHub data, how to ensure GitHub data accessibility and recoverability even during serious failures, and meet the best backup and Disaster Recovery practices. Check our short guidelines on how to protect your source code, thousands of hours of work (and money), and ensure an uninterrupted development process.

Why backup GitHub data?

If you've ever asked yourself this question, you can probably imagine how much it will cost your business to lose your source code hosted on GitHub. Hard to imagine these numbers? Let's try differently: how much does one hour of GitHub downtime cost your company? Now let's take a look at the most common threats to your repositories and metadata.

But before, let’s mention that GitHub follows the so-called Shared Responsibility Model according to which the company is responsible for infrastructure-level security while data protection of the single account stays among the user’s duties. That’s why even GitHub itself recommends having third-party backup software in place.

Outages - an unpredictable threat?

Believe us or check it on yourself but longer or shorter outages occur on GitHub on a regular basis. Only mentioning March 2022, we could spot the outages that affected about 73 million users.

There was a series of outages that GitHub explained as an issue due to the “health of their database”. Most of the complaints were connected to push and pull requests that developers failed to complete. Keith Ballinger, GitHub’s Senior Vice President of Engineering even posted a blog post saying “We know this impacts many of our customers’ productivity and we take that seriously.” Finally, GitHub has made improvements to its MySQL database cluster to address this issue and eliminate the further possibility of failure. But… who knows when another similar problem will appear on the horizon.

Long-lasting GitHub outages result in limited or disabled access to the account and data. This also results in high downtime costs for your business. In such situations, GitHub backup should enable you to instantly restore your entire GitHub environment to another git hosting service (i.e. Bitbucket or GitLab), to your self-hosted GitHub or local machine, and let your team work uninterruptedly.

If you want to stay up to date, we recommend monitoring the GitHub status page for failures and incidents updates.

Human errors - insider threats

Human error is another problem that is difficult to control but highly probable - it is considered the most common cybersecurity threat of all time. It can be an intentional (i.e. malicious activity of an ex-employee) or unintentional mistake that could lead to consequent failures and data breaches. I think we can agree with the GitGuardian team here - secret exposure is a great human mistake example. What are the others? Let’s name just a few:

branch deletion
old repository deletion
push force to master
losing/or not having a local copy

Can you imagine that according to TechCrunch even a GitLab sysadmin accidentally deleted a folder containing nearly 300GB of live production data making service unavailable for long hours? If it can happen to the biggest ones, it can happen to anyone.

Cyberattacks and ransomware - the rising threat

Do you know that ransomware attack attempts happen every 11 seconds? Recently, Dropbox suffered a data breach as a result of a phishing attack. Bad actors gained access to credentials, data, and other secrets inside their internal GitHub repositories. Only in the last few months, the list of companies that fell victim to attacks on GitHub repositories, and as a result of a data leak, includes brands such as Toyota, Uber, Samsung, Twitch, and more.

We talked about how GitHub enterprise backup helps reduce the scale of ransomware attacks and their effects during a joint GitProtect and GitGuardian webinar, the recording of which you can still watch on YouTube.

How to backup a GitHub repository

Once we understand the importance of automated backup in terms of all mentioned threats, we need to build a reliable backup and restore strategy that will allow us to restore data without affecting workflow continuity. Here we have two options. First - we can write our own, internal GitHub backup script and delegate someone from our team to monitor it on a daily basis. As it might seem a little investment of resources, in a long-term perspective it turns out to be both time and money-consuming. Oh, and did we mention we’ll need to test our restore strategy?

The second option is to use an automated, professional, third-party GitHub backup and Disaster Recovery software, like GitProtect, which enables you to set a backup policy on schedule and gives access to many professional security features.

Now, let’s check how to set up an efficient GitHub backup policy and what features are a must.

What data to include in your GitHub backup?

To get full assurance that the GitHub organization is secure and has the strongest protection, an enterprise should consider including all their GitHub repositories and metadata in their backup policy. To be precise, the backup should cover all the repositories, wiki, issues, projects, milestones, pipelines, issue comments, pull requests, deployment keys, webhooks, labels, pull request comments, or even Large File Storage (LFS).

Moreover, the backup software should permit the company to create different custom backup plans to meet the enterprise’s needs, workflow, structure, and safety requirements.

Unlimited retention

GitHub provides its customers with limited retention for deleted data - up to 90 days for public repos and a maximum of 400 days for private ones. Though, what if an organization needs that data for a longer period of time to meet its legal or security compliance requirements? For example, to meet SOC 2 or ISO 27001 standards? GitHub backup software should ensure you with unlimited retention, which you can use even to archive old, unused repositories for future reference, overcome GitHub storage limits, restore data from any point in time and keep your data as long as your legal recommendations require so.

Ransomware Protection

We have already mentioned that ransomware is one of the main threats the DevOps world is facing nowadays so it is worth considering a backup software equipped with some ransomware protection package. This should include immutable storage that prevents data in copies to be modified or erased.

What else? Take a look if your backup provides you with AES encryption (in flight and at rest) using your own encryption key, Data Center region of choice (for compliance purposes), and complete Disaster Recovery technology.

3-2-1 backup rule - multi-storage and replication

GitHub backup should follow the best “traditional” backup practices, and undeniably, the 3-2-1 backup rule is one of them. It states to maintain at least 3 copies of the data, keep 2 of them stored at separate locations, including 1 off-site.

To achieve this, your repository and metadata backup software should allow you to replicate the copy and add multiple data stores - both on-premise and cloud. Ideally, you can use the storage you already use, whether it's AWS Storage, Azure Blob Storage, Google Cloud, NFS, SMB, local disk resources, or others.

Backup monitoring

As mentioned before, one of the biggest pain points when it comes to GitHub backup scripts is… very time-consuming management. Third-party software has the advantage that it provides organizations with a central management console and intuitive data-driven dashboards. It should be easy to add additional admins, set roles and grant permissions to have more control over access yet share responsibilities of data protection among the team.

All automatic tools for ongoing monitoring are also important - audit logs, Slack notifications, e-mail reports with the most important data on backup processes, or compliance reports for the purposes of audits and security controls.

Proper backup monitoring can empower the DevSecOps team with appropriate control and give them the possibility to react immediately to any problem related to data protection, backup, and restore.

Backup as part of the CI/CD process

The desirable DevSecOps approach is based on the need to integrate security measures throughout the entire process of software development. Let’s consider a backup, which enables you to quickly roll back to the previous version of the code under any circumstances - whether it is a human mistake or any other event of failure. Including backup in a well-structured CI/CD process ensures flawless and predictable delivery. It is a “set and forget” process to ensure your peace of mind.

Disaster Recovery - Warranty of DevOps continuity

And a final thing… backup is useless if you don’t have the possibility to recover your data fast. This is again one of the biggest downsides of backup scripts and DIY methods - if you need to restore the data, you need to write another script with no guarantee it will work. The advantage of the software is the fact that it guarantees Disaster Recovery technologies in case of any scenario (and as far as you know from this article, the list of them is pretty extensive).

In the event of failure, service downtime, or cyber-attack you should be able to restore your entire GitHub environment to the same or a new GitHub account, to another git hosting service provider - Bitbucket or GitLab (in case of GitHub’s downtime or migration need) or to your local machine as a file. The main goal of Disaster Recovery is to ensure your company with uninterrupted DevOps processes, guaranteeing the shortest possible downtime and don’t risk financial and reputation loss.

It is also important to have a quick data recovery at hand in everyday work. Here comes a granular recovery of repositories and only selected metadata that enables you to get quick access to the data you want without the need to restore the entire GitHub environment. With point-in-time restore, you can restore data from any moment in time, from hours, a few days, or months ago.

Conclusion

DevOps security has ceased to be the responsibility of a few security specialists. More and more modern organizations aim to engage all team members and stakeholders to collaborate and proactively address security issues before software is developed and deployed. In terms of the growing cyber threat landscape and “shifting left” approach, GitHub backup of source code, as the most valuable Intellectual Property, should be considered as a key security measure to implement. But security works as a complete organism - so let's remember secret scanning, detection of security vulnerabilities, bugs, and more.

Security-first and “everyone is responsible for security” approach helps you to ship on time, increases developer productivity and as a result, provides your customers with a better and more secure experience.

Try GitProtect 14-days for free