What is the best solution for removing duplicate Objects from an Array?

(check out my blog)

There are many solutions to this problem but I wouldn't say that one is better than the others.

In this article I will just go through 2 approaches:

1. Using the standard built-in object "Set"
2. Using the method "reduce()" of Array ( Array.prototype.reduce() )

Set

From MDN Web Docs:

Set objects are collections of values. You can iterate through the
elements of a set in insertion order. A value in the Set may only
occur once; it is unique in the Set's collection.

Equality comparison

It looks like that Set it exactly the tool we need, but what does it mean "may only occur once"?

According to the documentation, Set uses the SameValueZero algorithm for the value-comparison operations, which means that it can determine whether two values are functionally identical in all contexts (0 and -0 are considered equal).

In other words, it's very similar to "===" (strict equality) with one exception: comparing NaN with NaN would return a truthy value.

Basic use case

Let's assume that we have this array:

const myArr = ['a', 'b', 'c', 'b', 'b', 'd'];

and we want to remove the duplicates.

Since the Set() constructor accepts an iterable as parameter (new Set([iterable])) and returns a new Set object, we can do the following:

    const mySet = new Set(myArr); 

mySet is now an instance of Set containing the following values:

'a', 'b', 'c', 'd'

Since the expected result we were looking for is an Array, we still have to convert the Set back to an Array.
We can easily perform this task spreading (...) the mySet values in a new Array:

const uniqValuesArray = [...mySet]; // [ 'a', 'b', 'c', 'd']

That's it :)

Complex object use case

The original question was: What is the best solution for removing duplicate objects from an array?
In the previous example we just used some used some string values.

Let's try to use this Array of Objects:

   let myArr = [
       {id: 1, name: 'Jack'},
       {id: 2, name: 'Frank'},
       {id: 1, name: 'Jack'},
       {id: 3, name: 'Chloe'}
    ];

We could try to use the same approach and create a new Set(myArr) from myArr, but in this case the comparison algorithm will consider every element of myArray unique, since the "SameValueZero algorithm" doesn't perform a deep object comparison:

    {id: 1, name: 'Jack'} === {id: 1, name: 'Jack'} // false

But what if we "prepare" our data and transform the object into something that fits better with the algorithm?

Let's create a new Array and fill it with the JSON-serialized version of the Objects:

    let myArrSerialized = myArr.map(e => JSON.stringify(e));

So we will have:

    ["{\"id\":1,\"name\":\"Jack\"}",  "{\"id\":2,\"name\":\"Frank\"}",  "{\"id\":1,\"name\":\"Jack\"}",  "{\"id\":3,\"name\":\"Chloe\"}"]

Where:

    "{\"id\":1,\"name\":\"Jack\"}" === "{\"id\":1,\"name\":\"Jack\"}" // true

Great. Now we have an array of values that fit with our purpose and the default Set's comparison algorithm.

Now we can continue as we did in the previous example:

    const mySetSerialized = new Set(myArrSerialized);

    const myUniqueArrSerialized = [...MySetSerialized];

But we need a new step at the end: we have to transform the serialized objects back to Objects:

    const myUniqueArr = myUniqueArrSerialized.map(e => JSON.parse(e));

That's it again :)

Summarising in a function

    const removeDuplicatesFromArray = (arr) => [...new Set(
      arr.map(el => JSON.stringify(el))
    )].map(e => JSON.parse(e));

Array.prototype.reduce()

The "reduce()" approach is also a good practice.
In the following example we consider "duplicates" two objects that share the same value of a specific key.

Let's work with this Array:

    let myArr = [
       {id: 1, name: 'Jack'},
       {id: 2, name: 'Frank'},
       {id: 3, name: 'Jack'},
       {id: 4, name: 'Chloe'}
    ];

The value Object {id: 1, name: 'Jack'} and {id: 3, name: 'Jack'} have different ids but the same name's value. That's why we consider them duplicates and we want to keep just the first of them.

Reducer

How Array.prototype.reduce() works isn't part of this post. If you don't know it, I recommend you to have a look to the documentation

This will be the reducer:

    const reducer = (accumulator, currentValue) => {
      if(!accumulator.find(obj => obj.name === currentValue.name)){
        accumulator.push(currentValue);
      }
      return accumulator;
    };

Basically we perform one simple check:

    !accumulator.find(obj => obj.name === currentValue.name)

We iterate over the given array.
Then, element by element, we check if we have already pushed in the accumulatoran Object with the same value of the name property.
If no element matches the condition we push the current element in the accumulator otherwise we just skip the step.

So we just have to apply the reducer we just created to the Array and initialize the accumulator with an empty array:

    myArr.reduce(reducer, []));

Summarising in a function

    const removeDuplicatesFromArrayByProperty = (arr, prop) => arr.reduce((accumulator, currentValue) => {
      if(!accumulator.find(obj => obj[prop] === currentValue[prop])){
        accumulator.push(currentValue);
      }
      return accumulator;
    }, [])

    console.log(removeDuplicatesFromArrayByProperty(myArr, 'name'));

Let's combine both these approaches

As Andrea Giammarchi (ehy, thank you!) pointed out, it's even possible to combine both solutions!
A premise is needed.
As a second paramater, Array.prototype.filter() accepts the value to use as this when executing callback.

let newArray = arr.filter(callback(element[, index[, array]])[, thisArg])

Now we can explore the new solution:

const by = property => function (object) { 
  const value = object[property]; 
  return !(this.has(value) || !this.add(value));
};

const myFitleredArr = myArr.filter(by('name'), new Set);

Let's read it line by line:

const by = property => function (object) { 

This is test function we'll pass to a filter() method in order to test/filter each element of the array.

 const value = object[property];

Assign to "value" the value of the given object's property.

 return !(this.has(value) || !this.add(value));

Return true to keep the element, false otherwise.
Remember that "this" in our example will be "new Set".
If the Set doesn't already have the given value it will return true and add the value to the collection.
If the Set already has the given value, then it won't keep the element.

In this example is possible to reuse the given Set, the one we pass as second parameter to the method filter().

If you don't need to reuse it you can create a new Set everytime:

const by = property => {
  const set = new Set;
  return obj => !(set.has(obj[property]) || !set.add(obj[property]));
};

About this post

I'm am running a free JavaScript Learning Group on [pixari.slack.com] and I use this blog as official blog of the community.
I pick some of the questions from the #questions-answer channel and answer via blog post. This way my answers will stay indefinitely visible for everyone."

If you want to join the community feel free to contact me:

• @pixari on Twitter
• [e-mail]

Comments

[Andrea Giammarchi]

You could also combine both solutions.

const by = property => function (object) {
  const value = object[property];
  return !(this.has(value) || !this.add(value));
};

const myFitleredArr = myArr.filter(by('name'), new Set);

This way you could reuse over and over the given Set, but you could also make it a one-shot each time.

const by = key => {
  const set = new Set;
  return obj => !(set.has(obj[key]) || !set.add(obj[key]));
};

[Raffaele Pizzari]

This is a very good hint.
I'll edit soon the post adding this solution with some explanations.

Thank you, Andrea!

[Raffaele Pizzari]

Done :)

[Andrea Giammarchi]

Thanks, but there is a typo apptets instead of accepts, and I think you should explain the one-off approach too, where you don't need to pass any Set.

Passing new Set is not super handy if you never reuse that Set, so that in such case not passing a second argument to filter(...) provides a better UX.

[Raffaele Pizzari]

Thanks again, Andrea!

[Vince Varga]

Hey Raffaele, Great post!

As this post is quite detailed and the main audience is people who learn JavaScript, I just wanted to mention about the JSON method that, for JSON. stringify() , the order matters.

JSON.stringify({id: 1, name: 'Jack'}) === JSON.stringify({id: 1, name: 'Jack'})
true
JSON.stringify({id: 1, name: 'Jack'}) === JSON.stringify({name: 'Jack', id: 1})
false

Though I'm sure you know it, it might be relevant to readers (or on the other hand, it might be too much information and out of scope for the article :))

[Raffaele Pizzari]

Thank you for pointing this out!

I'm sure it would be pretty useful for developers who are learning JS and aren't very confident with string comparison.

Thank you again :)

[Aditiya Prayogo]

You could also use array.find

const duplicateNote = notes.find((note) => note.title === title)