DEV Community: Felipe

Dealing with duplicates

Felipe — Tue, 09 Apr 2024 22:02:19 +0000

I've been working on a inventory management app and recently ran into an oversight in the initial rollout of the database.

This one oversight landed me with a total duplicate count of 186 records stemming from what should have been 82 unique records. So trying to do the math, it's not clean case of two duplicates per item. Which meant that trying to consolidate based on latest modified or vice-versa would not work as some of the latest duplicates of a given part contained missing or bad data and others didn't, in other cases up to 4 duplicates would be found - making it complex to determine which one to pick programmatically.

After taking some time to think through ways to fix this I landed on creating a few reports in a way that grouped all the duplicates and showed what was changed between them, and with a recovery strategy in hand, drop all the records with duplicates, add the uniqueness constraint and re-insert the records with the reports in hand and ability to check the inventory physically.

Even though it was a costly mistake on my end, it was the best way to guarantee the data would be accurate.

First step: Backups

I'm using a managed Postgres instance on Heroku for the prod database and luckily it runs backups periodically. But it's not enough. So I decided to create an cron job to run a daily manual backup (the database is quite small at this point in time).

#! /bin/bash

echo "Running database backup"

backup_dir="/Users/felipe/backups/<client>-db-backups"
latest_backup=$(ls -t "$backup_dir"/*.dump | head -n1)

if [ -n "$latest_backup" ]; then
    echo "Previous backup found"
    last_modified=$(date -r "$latest_backup" "+%m-%d-%Y %H:%M:%S")
    echo "Created at: $last_modified"
else
    echo "No existing backup was found."
fi

echo "Running backup..."

backup_filename="$backup_dir/<client>_$(date +%m-%d-%y).dump"
heroku pg:backups:capture -a server
heroku pg:backups:download -a server -o "$backup_filename"

So breaking it down:

Gab the latest backup, if any, from the backup directory
If it exists, proceed to print a message declaring a backup exists and when it was created to stdout
If it doesn't exist, it will print a message declaring there are no backups found
Capture and download a backup from the prod instance

Finally, add the job to the cron table

$ crontab -e 

0 3 * * * /bin/bash /path/to/the/script.sh

Second step: Write the update script

So this part is fairly simple but a lot can go wrong.

First I ran a few queries to ensure what I was about to do was targeting the right data:

-- 1: find the total count to get an idea of how much data we're talking about
SELECT COUNT(*)
FROM (
    SELECT
        pn,
        COUNT(pn)
    FROM
        sometable
    GROUP BY
        pn
    HAVING
        COUNT(pn) > 1
);

-- 2: unpack the previous query and get an idea of much duplicates per item
SELECT pn, count(pn)
FROM sometable
GROUP BY pn
HAVING COUNT(pn) > 1;

-- 3: inspect the duplicate data
SELECT *
FROM sometable
WHERE pn IN (
    SELECT pn
    FROM sometable
    GROUP BY pn
    HAVING COUNT(pn) > 1
)
ORDER BY pn;

Cool, after looking over the result sets I felt fairly confident about what I was about to delete as well as took the opportunity to export these reports as they would come in handy in the final step.

DELETE FROM sometable
WHERE id IN ( 
    SELECT *
    FROM sometable
    WHERE pn IN (
        SELECT pn
        FROM sometable
        GROUP BY pn
        HAVING COUNT(pn) > 1
    ) 
    ORDER BY pn
  );

-- Let's try to prevent this from happening again
ALTER TABLE sometable
ADD CONSTRAINT unique_col UNIQUE (pn);

In part 2 I will work on making changes to both the client and API to improve the UX and API around handling a possible duplicate insertion gracefully. Also, if you'd like to share any similar stories please do.

Learning React Part 1: Stateless Components

Felipe — Sat, 12 Sep 2020 21:07:06 +0000

How does defining a stateless component in React differ from Vue? This article will go over my initial observations and is the first in a series that will cover my experience learning React having worked primarily with Vue in the past.

Defining stateless components in React

import React from 'react';

const myComponent = props => {
    return (
        <div className="myComponent">
            <p>props.myProp</p>
        </div>
    );
}

export default myComponent;

Just like in Vue, React has more than one way to define components. Being that most SPAs are basically component trees and the most common type of components used in building an application's UI are presentational or stateless. Stateless components seem like a good place to start.

The most common way to define stateless components in React is using functional component syntax. In essence, you are simply defining and exporting a function which will then be passed to ReactDOM.render().

The function encapsulates the component in the return statement - in JSX, which is just JavaScript at the end of the day. This can be done without the enclosing parens if it's a single line or between parens for multiline/nested markup.

    const component = () => return <div>Hello world!</div>

    const component = () => return (
        <div>
            <h1>Hello World</h1>

            <p>This is multiline JSX</p>
        </div>
    )

Behind the scenes React is using React.createElement() to render the actual markup in the DOM. This function takes three arguments, the tag type as a string, an object containing attributes - if any, and lastly the text/content. This is a contrived example. You can have multiple nested children in that element so you can see how it may get a bit cumbersome to describe markup in this fashion.

Attributes are handled a bit differently than Vue SFCs. Since you are not really working with HTML but JSX, camelCase is used to work with attributes.

So class="myComponentClass" is className="myComponentClass in React.

The last step is to export the component itself so that it may be used by other components.

Since this component will not hold any state, it is important that we pass the props argument in the arrow function assigned to the component to access data the consumer may pass down to it.

Defining stateless components in Vue SFCs

<template functional>
    <div class="myComponent">
        <p>{{myProp}}</p>
    </div>
</template>

<script lang="ts">
import Vue from 'vue';
import {Component, Prop} from 'vue-property-decorator';

@Component
export default class extends Vue {
    @Prop
    myProp: string;
}
</script>

Vue uses a different approach with its Single File Component syntax, which allows you to define the markup, scripts, and styles in one file, although this is simply a convention. You can also define SFCs in separate, corresponding, files and reference the styles and script in the .vue template file just like you would in any HTML file.

Mark up is defined in the <template></template> tag and you can directly access and manipulate props with interpolation using the double mustache {{}} which binds the data. It is important to either pass the functional keyword after template in the opening tag in SFCs or as a prop when defining the component with the regular syntax:

Without JSX

Vue.Component('my-component', {
    functional: true,
    props: [myProp],
    render: (createElement, context) {
        return createElement('p', props.myProp)
    }
})

With JSX

Vue.Component('my-component', {
    functional: true,
    props: [myProp],
    render: (h) => (
            <p>props.myProp</p>
        )

We then define the component in the script tag. There are multiple ways to do this. I prefer class-based components using TypeScript with the vue-property-decorator library which allows me to define the component using decorators that Vue can then understand and use in the compilation process.

@Prop allows us to define prop attributes on the component, and also allows for defaults to be defined if necessary.

@Component is actually part of the Vue Class Component library which vue-property-decorator depends heavily on. This decorator allows us to use class-style syntax to define our components.

These are my preliminary thoughts on stateless components as I am going through the process of learning React. Subsequent posts will go over other aspects, similarities, and differences.