AWS DMS and Prefect: The Key to Building a Robust Data Warehouse

Fady khallaf — Sat, 14 Jan 2023 22:46:20 +0000

AWS DMS (Database Migration Service) is a fully managed service that makes it easy to migrate data from multiple sources to a variety of targets, including Amazon RDS, Aurora, and Redshift, as well as other databases on-premises or in the cloud. One of the key features of DMS is the ability to capture data changes in real-time and replicate them to the target database.

There are two main ways to use DMS to capture data changes:

Full load and change data capture (CDC)

This method involves performing a full load of data from the source to the target, and then subsequently capturing and replicating only the changes made to the source data. This method can be used when migrating data from a source database that supports CDC, such as Oracle, SQL Server, or MySQL.

To set up full load and CDC in DMS, you need to create a replication instance, a source endpoint, and a target endpoint. You can then create a replication task and configure it to use the full load and CDC method. DMS will automatically capture and replicate changes made to the source data in real-time.

Replication using CDC-enabled API

This method involves using a CDC-enabled API, such as AWS AppSync or AWS Lambda, to capture data changes and send them to DMS. This method can be used when migrating data from a source database that does not support CDC, such as MongoDB or PostgreSQL.

Using DMS to capture data changes in real-time can save time and effort when migrating data, as it eliminates the need to manually track and replicate changes. Additionally, it ensures that the target database is always up-to-date with the latest changes in the source database.

In the following paragraph, I will share with you how we used AWS DMS to capture data changes using binlog from our MySQL database and store it in an S3 bucket.

First, to capture binlog in MySQL database, you should check that the binlog format is of type ROW in the parameter group so that you can find the data changes like any (Insert, Updated or Delete) process in the binlogs.

Second, use AWS DMS to move binlogs from the RDS to an S3 bucket by setting the DMS source endpoint to the RDS and DMS target point to the S3. Now all your binlogs are collected in the S3 bucket, so you can use any data orchestration tool like (Prefect, Dagster, Airflow, or any other data engineering pipeline tool)

In our use case, we used Prefect, which is a modern, open-source workflow management system that can help you schedule, execute, and monitor your data pipeline tasks.

With Prefect, you can easily create workflows that include tasks such as data extraction, cleaning, and loading. You can set up your data pipeline to run on a schedule, such as Hourly, daily, or weekly. It also offers built-in monitoring and alerting, so you can easily keep track of the status of your pipeline and troubleshoot any issues that may arise.

To use DMS and Prefect together, you can create a Prefect task that triggers a DMS migration, and then chain other tasks in the workflow to perform additional actions on the data (Binlogs in the S3), such as cleaning or transforming it before loading it into Redshift as a staging schema.

And then, we used another Prefect workflow to do data validation and transformation on the staging schema to load it into the data warehouse presentation layer so that analytics tools and data science models can rely on this data.

In summary, AWS DMS and Prefect can be significant players in your journey to build your own data warehouse.

Cost Cutting AWS strategies

Fady khallaf — Fri, 13 Jan 2023 16:07:48 +0000

Amazon Web Services (AWS) is a popular cloud computing platform that offers a wide range of services to businesses of all sizes. However, as with any cloud-based platform, costs can quickly add up if not properly managed. In this article, we will discuss several strategies and steps that businesses can take to reduce their AWS costs and maximize their savings.

1. Utilize the Graviton processor:

The Graviton processor is a custom-designed processor created by AWS that is optimized for running cloud-native workloads. Utilizing this processor can lead to significant cost savings, as it is often cheaper than using other processors. To take advantage of this cost-saving opportunity, businesses should consider migrating their existing workloads to instances that use the Graviton processor, it’s highly recommend to migrate all your manged DB services to Graviton machines so if you have RDS, Elasticache, RedShift, .. etc migrate these managed serivces to Graviton machines.

2. Use Reserved Instances:

Standard Reserved instances allow businesses to commit to a certain machine family for a period of time, in exchange for a lower hourly rate. This can lead to significant savings, especially for businesses that have a consistent usage pattern. You can change machine size but not the family with Standard Reserved Instances, if you have any concern that you might want to change the family within your commitment you can go with Convertible Reserved Instances opion.
Tip: For RDS go with graviton machine first and then go with Reserved Instance option combine both options will cut your cost significantly.

3. Use Compute saving plans:

Saving Plans is the new pricing model from AWS it came with 3 options:-

EC2 Saving Plans
Compute Saving Plans
Amazon SageMaker Saving Plans

I am interested here to write about Compute saving plans allow businesses to commit a consistent amount of usage (measured in $/hour) for a 1 or 3 year term, in exchange for a lower hourly rate. The compute saving plan is ideal for businesses that have want to try this payment option with fewer constraints, as they can lead to significant savings.

4. Use GP3 for storage volumes instead of GP2:

GP3 is a newer generation of storage that offers higher performance and lower costs than GP2. GP3 also offers several features such as higher IOPS, lower latency, and more storage. To take advantage of this cost-saving opportunity, businesses should consider migrating their existing GP2 storage to GP3.

5. Workload Rightsizing

Use AWS Compute Optimizer to analyze your Amazon Elastic Compute Cloud (EC2) and Amazon Elastic Container Service (ECS) usage and provide recommendations to help you optimize your compute resources.

The service analyzes your usage data and provides recommendations to help you:

Right-size your EC2 and ECS instances based on your usage patterns
Identify and terminate underutilized instances
Schedule instances to turn off when they are not needed
Identify and move workloads to lower-cost instances
Identify opportunities to use reserved instances and savings plans

6. Shut down Non Prod workload in the Non business hours

AWS has service called AWS Instance Scheduler can be used to automatically stop instances and tasks outside of business hours, such as in the evening or on weekends, when they are not being used. This can help customers to reduce their costs by only running instances and tasks when they are needed.

7. Monitor and Optimize:

Continuously monitoring and optimizing your usage can help you identify areas where you can reduce costs. Use the Cost Explorer to identify the most expensive services and resources, and then take action to optimize or eliminate them.

In conclusion, These modifications can help you reduce and cut-down your AWS costs without any engineering overhead.

DEV Community: Fady khallaf