Developing a data loading tool from scratch

Aim: To Copy data from Database X to Database Y

Assumptions:

1. Large data may be in petabytes.
2. Application will be hosted in cloud(In house application)

My take:

1. Reading data from database
2. Writing into blob storage.
3. From blob to new database.
4. Configuration to keep track of source and destination mapping( Table, schema).
5. Error management (Abort/skip).
6. From 2-3, error reload mechanism from point of failure.
7. From 1-2, i am not sure about the error reload mechanism.

Can anyone share some best practices/thoughts

Comments

[Jeremy]

Hello @pogo420! I’m curious to know how you went about developing this tool. Would you mind sharing your experience, what worked well and what didn’t work well? Thanks!