Migrating on-premises block-based workloads to the cloud can be a daunting task, especially when the workload contains a large amount of data.
This is a follow-up to the site-to-site VPN setup I documented here https://dev.to/ashish_nair_d9b10ba4f8126/aws-site-to-site-vpn-using-strongswan-1olk .
For this lab, I wanted to combine the best of both worlds: keep the workload on-premises while using AWS as the destination for storage, backup and recovery — and keep the connectivity private using the VPN and AWS private endpoints.
The Goal:
My on-premise MariaDB server writes it's data to /var/lib/myql.
For this experiment, I wanted to explore two possibilities:
- Keep a copy of the block-based workload in AWS and use EBS snapshots for backup/recovery.
- Recover the block-based filesystem in AWS, attach it to an EC2 instance, and eventually move the compute workload to AWS.
I configured the Storage Gateway appliance to communicate with AWS through the private connectivity already established by my Site-to-Site VPN and used AWS private endpoints for the AWS-side connectivity.
This allowed me to keep the Storage Gateway traffic on private network paths rather than sending it over the public Internet.
The Final test:
Can I take the data that my on-premises MariaDB server was writing to, recover that storage in AWS, attach it to an EC2 instance, and verify that my database data is still there?
Why not just use mysqldump?
A very reasonable question is: Why not simply take a MariaDB/MySQL dump and restore it in AWS?
For a database migration, that can certainly be a much simpler approach.
However, that isn't what I wanted to experiment with here.
Let's assume instead that we have a very large block-based filesystem containing a large amount of data.
Rather than thinking about the contents of the filesystem as individual database objects or files, this experiment looks at the problem from a block-storage perspective.
That makes Storage Gateway a much more interesting technology to explore.
How does this work?
The interesting part of this setup is that MariaDB doesn't need to know that AWS is involved.
The Storage Gateway appliance exposes an iSCSI block device to the on-premises MariaDB server.
From the MariaDB server's perspective, it simply sees a block device
I scanned for the iSCSI target from the MariaDB server, logged into it, formatted the volume and mounted it as a normal filesystem.
The application therefore continues to interact with the filesystem normally.
Behind the scenes, the Storage Gateway handles the transfer of the data to AWS. For a cached volume, the gateway maintains local cache/upload-buffer storage and asynchronously transfers data to AWS.
This is what makes the experiment interesting: the application is working with a normal block device while the underlying storage is being integrated with AWS.
The Lab:
- I first created a Storage Gateway in AWS and selected the appropriate gateway configuration for the experiment.
- The next step required downloading the VMware appliance and starting it in my local VMware environment. The appliance provides the interface between my on-premises environment and AWS.
- The gateway appliance console looked something like this:
- I attached a 150G disk to the storage appliance. This storage is used by the gateway as part of the volume configuration and allows it to expose an iSCSI volume to the on-premises server.
- From the on-premises MariaDB server, I scanned for the iSCSI target exposed by the Storage Gateway appliance and logged into it.
- I formatted the newly discovered block device and mounted it on the MariaDB server.
- Once the filesystem was mounted, I generated some entries in the database. This gave me some known data that I could later use to verify whether the migration/recovery actually worked.
After the volume was being used by the on-premises server, I could see the corresponding volume through the AWS Storage Gateway console.
I then created a snapshot of the Storage Gateway volume. This was the point where I could verify that AWS had successfully received the data. However, a successful snapshot alone isn't enough. I wanted to prove that the data could actually be recovered.
- I used the snapshot to create an EBS volume and attached that volume to an EC2 instance.I then mounted the recovered filesystem on the EC2 instance.The final test was to inspect the MariaDB data.
The database entries that I had created while the workload was running on-premises were present in the recovered volume in AWS.
That gave me confidence that the complete path had worked.
This experiment demonstrated that I could take a block-based workload being accessed from my on-premises environment, expose it through Storage Gateway, create a recovery point in AWS, recover that storage as an EBS volume and verify the data from an EC2 instance.
The important part for me was that the storage abstraction was transparent to the application. MariaDB interacted with a normal filesystem and didn't need to know whether the underlying block storage was ultimately being integrated with AWS.
However, this experiment should not be interpreted as demonstrating an application-consistent MariaDB backup strategy. For production database workloads, database-native replication, AWS DMS, or an application-consistent backup procedure may be more appropriate depending on the RPO/RTO requirements.
Top comments (0)