DEV Community

Ben Halpern
Ben Halpern Subscriber

Posted on

How does your organization handle data backups?

Top comments (29)

Collapse
 
vyckes profile image
Kevin Pennekamp

Wait, do we need to backup data?! sh*t...

Collapse
 
nijeesh4all profile image
Nijeesh Joshy

If someone asks, just say its all in the cloud ( pointing your finger upwards ).

Collapse
 
ben profile image
Ben Halpern

Collapse
 
itsasine profile image
ItsASine (Kayla)

Collapse
 
picocreator profile image
Eugene Cheah • Edited

I point to our cloud provider "RDS" SQL solutions - they managed the day to day backups.

However, because I cannot get rid of the paranoid itch in me (what if AWS & GCP goes down?)

I do a full dump onto a dedicated company server in a local data center about every month or so (in Singapore, where I am at). That is independent of the previous months dumps.

Just in case.

Collapse
 
ferricoxide profile image
Thomas H Jones II

If you don't have legal locality-requirements, you could always do your backups to a replicated S3 bucket. That way, even if an entire region goes down, you'd still have recoverability. Then again, (and, again, if you don't have legal locality-requirements) you could probably also set up a job to directly replicate your RDSes from one region to another.

Collapse
 
picocreator profile image
Eugene Cheah • Edited

Yup - and it's quite awesome it can be automated with a few clicks (in the right places)


Btw: While this may sound quite dumb.

Where I am at. It is common to have disaster recovery plans which outline backups must not be the same company. You know just in case the entire account gets deleted, with data purged and all.

Because our critical data is not "that" huge - dumping it works well for me to comply with it (with encryption and all of course)

Thin foil hat, so that our auditors and investors can feel safe knowing that our data is safe in the event of WW3, zombie apocalypse, etc. We have a recovery plan for that!.

Nevermind the fact, that we do not have such a plan in place for food, water, zombies or people 😅

Collapse
 
gsogeek profile image
Matt Ballance

My background was IT/DevOPS/SecOPS before "inheriting" my way into development, so I got to influence our process when I came on board. Please excuse the wall of text and any misspelled wurds you see.

We re-did our whole backup strategy from a "whenever we thought about it" to an automated scheduled process with manual options for immediate needs:

On premise SQL data (MSSQL) has transaction logs snapped every 1/2-, 2-, or 4-hours depending on the database, and a full backup every night. I'm still working on convincing the people that control our MySQL server that we need a true replication scheme and backup strategy for that system.

We're finishing up moving our on-premise code repositories from an old TFS server to VSO which are also backed up to a cloud storage solution with rolling snapshots of all of the repositories every 4 hours.

Various on premise servers have a rolling differential backup done anywhere from every day to real-time depending on what kind of data it's got (the more mission-critical, the faster they get backed up).

The goal we set was to have point-in-time recovery on all of our mission critical data while still being able to protect everything else and not blow the budget out of the water. The changes have saved us on more than one occasion where the data loss was measured in seconds using this process instead of days in the previous.

If you made it this far, a bonus idea for you: every so often, take your backup file and try to restore it somewhere that's not important, like a dev server or test server. You'd be surprised how often the backup is corrupted or has some crazy quirk to get it to restore. 3AM the day of a 9AM release is not the time to discover this happened to you...

Collapse
 
jamonjamon profile image
Jaimie Carter

This sounds like a great strategy. Redundancy is so important. I also like the idea of spinning up the backed up data to see if the backup is right. I think that would be a critical element to any redundancy.

Collapse
 
rachelsoderberg profile image
Rachel Soderberg

I know we have some kind of automated process on our server that runs at a fairly regular interval - but I don't know enough about it to explain it.

Sheepish Emoji

Collapse
 
ferricoxide profile image
Thomas H Jones II

You'll know if it worked or not when disaster strikes. :p

Collapse
 
rachelsoderberg profile image
Rachel Soderberg

Oh it works, I've been the disaster that has struck a few times now. LOL

Thread Thread
 
xanderyzwich profile image
Corey McCarty

Anyone that hasn't been is either lying to you or too new for it to have happened yet

Collapse
 
ahferroin7 profile image
Austin S. Hemmelgarn

We actually don't handle many databases, and luckily have near certain periods of downtime when they're sitting idle. Given this and the fact that we're stuck dealing with software written by idiots who didn't understand the concept of automation, we just back up the databases like we do any other files on the systems (albeit while forcing a commit just prior to the backup), by backing them up using Borg (with the data that needs backed up accessed remotely by the backup server when dealing with Windows systems) and then synchronizing the Borg repository off-site using rclone.

Collapse
 
simbo1905 profile image
Simon Massey

we backup our database...? well we currently have a two-node mysql compatible aurora cluster on aws so that we have a hot standby with automatic promotion when the master goes off-line. we still use backups in case we damage things and the damage gets replicated to the slave. backups are something you simply specify on the aws dashboard. we are then relying up on aws being able to not loose all the backups at the point where we damaged the database which seems okay.

Collapse
 
yaser profile image
Yaser Al-Najjar

I spent too much time to write a serverless lambda function that goes into the server and upload the backup into S3

github.com/coretabs-academy/postgr...

Guess what?
It didn't work in production, so I keep doing backups manually every once in a while 😂

Collapse
 
alexanderalemayhu profile image
Alexander Alemayhu

For legacy systems we use scripts run via cron jobs that dump databases (this one sucks due to disk space potentially being used up during backup) and more recently automated snapshots via snapshooter.io

For dev machines an external hard drive using time machine works great :)

Collapse
 
ferricoxide profile image
Thomas H Jones II

Depends on where in the organization. In the legacy datacenters, "NetBackup". For those who've moved to the cloud - where the NetBackup domain doesn't reach and where architecting the connectivity to allow a media server to effectively operate there, it's "self-service".

Early on, we provided a set of simple reference scripts to show the cloud-users, "here's how you can use cron/scheduled tasks to leverage the CSP's native capabilities" (i.e., "volume snapshots" to provide at least crash-consistent recovery options). But, mostly, we've been encouraging those same users to leverage CSP options that include baked-in data-backup options.

Collapse
 
crimsonmed profile image
Médéric Burlet

For our part we have cloud backups for our client data and also like to keep a paper copy for contract and other documents. We wouldn't want a google drive crisis to impact our business for instance.

For code we use github organization and keep a copy of each projects code as well as an off site raspberry pi git server. Which itself makes code copies on multiple hard drives.

We also do manual backups of our computer to hard drives which are then stored away in case of emergency in a fireproof small safe. (non network connected backups are great in case of ransom wares for instance)