Removing Duplicate Objects from an Array

One of the most common data structure that web developers encounter is an array of objects. Many times the data provided are not formatted or structured to accommodate for the needs of the user. As developers we will typically be required to manipulate the data so that it's tailor fit for our own use case. In this blog I will be discussing a creative solution that I came across online that is used to remove duplicate objects in an array.

To start off, let's say we have a given data structure shown below:

const students = [
  {id: 12, fName: 'John', lName: 'Smith'},
  {id: 75, fName: 'Anthony', lName: 'Smith'},
  {id: 100, fName: 'Mary', lName: 'Jane'},
  {id: 12, fName: 'Jane', lName: 'Doe'},
]

In this data structure the id with the value of 12 is repeated twice. What if we only want to show 1 object with an id of 12 in that array? This can be accomplished by using a combination of common Javascript methods which includes: reduce, set, values, and the spread operator.
In order to understand how these methods can be used in combination with one another it is advised that you have have a basic understanding of each method which are covered below. If you are already familiar with these methods you can skip to the Solution section.

Reduce

The reduce method is used on an array and reduces it to a single value. An example of the syntax for this method is shown below:

const array = [1,2]
array.reduce(function(accumulator, currentValue){return accumulator + current value}
, initialValue)

As shown above, the reduce function takes in 2 parameters with one being the callback function that is going to be called on each element of the array as it iterates through and the second parameter being the initial value(the value that will be assigned to accumulator on the first iteration). As the callback is being called on each iteration the result of the callback will be stored in the accumulator value which will then be used in the subsequent iteration until every element in the array has been iterated through

For this example, if we set the initial value to 10, on the first iteration using 1 as the current value the callback function will return 11. So accumulator = 10 + currentValue = 1 will return 11. On the next iteration 11 will be stored in the accumulator parameter and currentValue will be 2 and the callback will return 13 since 11 + 2 = 13.

In summary, we started off with an array with 2 values and given a starting value of 10 the reduce method returned a single value of 13. In addition, keep in the mind that when using reduce, single value does not mean that it has to be a single number, it can also be a single object(more on this later).

Set

When this method is called on a map object it will populate the object with a unique key value pair. If the map object already contains a key that you're trying to add it will override that key value pair with the current key value pair to prevent any duplicate keys. Below is an example of adding key value pairs to an empty map using set.

const map1 = new Map();

map1.set('a', 1);
map1.set('b', 2);
map1.set('c', 3);
map1.set('a', 3);

Notice how in the above example that I tried to add a duplicate key value of a:3. Since this was the last key value pair added to the object and the key a already exists in the map object, if I were to console.log(map1) key a will only show up once and the value for the key will be 3 since set overrides any duplicate keys.

Values

When the values method gets called on an object it will return an array with just the values. See below for an example:

const object1 = {
  a: 'somestring',
  b: 42,
  c: false
};

console.log(Object.values(object1));
// expected output: Array ["somestring", 42, false]

Spread Operator

The spread operator allows us to copy an existing array into another array or object. See below example:

const numbersOne = [1, 2, 3];
const numbersTwo = [4, 5, 6];
const numbersCombined = [...numbersOne, ...numbersTwo];

Solution

Now that we have a basic understanding of these 4 methods we can now dive into the solution of our original problem which was to remove duplicate objects within an array. Recall that our students array is as shown below:

const students = [
  {id: 12, fName: 'John', lName: 'Smith'},
  {id: 75, fName: 'Anthony', lName: 'Smith'},
  {id: 100, fName: 'Mary', lName: 'Jane'},
  {id: 12, fName: 'Jane', lName: 'Doe'},
]

Using the reduce method on the above array we can write:

const uniqStudents = students.reduce((map, obj) => {_code goes here_}, newMap());

So we are calling the reduce method on the students array and passing 2 parameters into reduce. The first parameter is the callback function that is taking in another 2 parameters, map and obj with map being the accumulator value and obj being the current value. The second parameter that is being passed into reduce is the initial accumulator in this case is newMap() an empty map object.

Now let's add in the code inside the callback function brackets.

const uniqStudents = students.reduce((map, obj) => {return map.set(obj.id, obj)}, new Map());

Let's unpack what is happening here.

The reduce method is iterating through each object in the initial students array given the initial accumulator value being an empty object.

On the first iteration we are adding a new object to the empty map object using .set and specifying that the key for the object being added is obj.id and value being the entire original object.

Once reduce iterates through every object in the original, the empty map object will now contain:

Notice how the key for id:12 only shows up once now. This because we used the .set method which overrides any duplicate keys with the latest key value pair. Now that we've got rid of the objects with duplicate Ids the last thing to do is to grab the values of each object and put them into a new array using the .values() and the spread operator. The code will look something like this:

const uniqStudents = [...students.reduce((map, obj) => {return map.set(obj.id, obj)}, new Map()).values();]

With the result looking like this if we console.log(uniqStudents):

In summary, we were able to reduce an array of objects into a single value(in this case is a single object with other objects nested within) and used the set method to remove any objects with duplicate keys that are nested in that single object. After removing the objects with duplicate keys we then use the values method to grab the values of each object and place it into a new array using the spread operator.

Conclusion

Just from this one scenario of removing duplicates we've utilized a bunch of common Javascript methods which really goes to show the power of basic Javascript functions when used in combination with one another! What are some other ways to remove duplicate objects in an array? Let's discuss!

Resources:

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_objects/Object/values
https://www.w3schools.com/react/react_es6_spread.asp
https://www.youtube.com/watch?v=5JFJTGZ4gHQ

Comments

[Jon Randy 🎖️]

const students = [
  {id: 12, fName: 'John', lName: 'Smith'},
  {id: 75, fName: 'Anthony', lName: 'Smith'},
  {id: 100, fName: 'Mary', lName: 'Jane'},
  {id: 12, fName: 'Jane', lName: 'Doe'},
]
const uniqStudents = Object.values(students.reduce((l,s)=>(l[s.id]=s,l),{})

Map isn't necessary.

Interestingly, this solution is orders of magnitude faster the the original in Firefox, but the result is reversed in Chrome. Different engines, different optimisations.

[KevinZ-CS]

Agreed! Thanks for the read Jon and for sharing your analysis!

For future readers feel free to take a look at Jon's analysis where he uses a simple object rather than a map object. Both solutions will remove duplicate objects but as Jon mentioned will yield different optimizations depending on what engine you use.