The latest articles on DEV Community by sunil (@jdisunil). https://dev.to/jdisunil https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1229129%2F8dc59dc9-6be7-49fa-bdf4-c6c7a8ef62b6.png https://dev.to/jdisunil en sunil Fri, 08 Dec 2023 15:06:57 +0000 https://dev.to/jdisunil/synapse-pyspark-delta-lake-merge-scd-type2-without-primary-key-1cjh https://dev.to/jdisunil/synapse-pyspark-delta-lake-merge-scd-type2-without-primary-key-1cjh <p>*<em>Problem<br> *</em><br> I have a set of rows coming from previous process which has no primary key, and the composite keys are bound to change which are not a good case for composite key, only way the rows are unique is the whole row( including all keys and all values). I need to implement the SCD type2 on this data. The environment is Synapse pyspark, using delta lake Merge command and more.</p> <p><strong>how I tried</strong><br> Using row hash: In this case the challenge without primary/composite key is to find which rows have changed/updated. With any updated values the row hash is changing and resulting into new row.</p> <p>please suggest how this problem can be solved. If you have any questions on this, please write back. </p>