The latest articles on DEV Community by Arnab (@pogo420). https://dev.to/pogo420 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F500680%2Fb2571909-8abf-4d82-a79c-f17061e8399b.jpg https://dev.to/pogo420 en Arnab Mon, 13 Dec 2021 11:36:39 +0000 https://dev.to/pogo420/developing-a-data-loading-tool-from-scratch-2f5l https://dev.to/pogo420/developing-a-data-loading-tool-from-scratch-2f5l <p>Aim: To Copy data from Database X to Database Y</p> <p>Assumptions:</p> <ol> <li>Large data may be in petabytes.</li> <li>Application will be hosted in cloud(In house application)</li> </ol> <p>My take:</p> <ol> <li>Reading data from database</li> <li>Writing into blob storage.</li> <li>From blob to new database.</li> <li>Configuration to keep track of source and destination mapping( Table, schema).</li> <li>Error management (Abort/skip). </li> <li>From 2-3, error reload mechanism from point of failure.</li> <li>From 1-2, i am not sure about the error reload mechanism.</li> </ol> <p>Can anyone share some best practices/thoughts</p> design systems cloud