DEV Community

Ankit Kamboj
Ankit Kamboj

Posted on

Javascript remove duplicate from array of objects

const seen = new Set();
const arr = [
  { id: 1, name: "test1" },
  { id: 2, name: "test2" },
  { id: 2, name: "test3" },
  { id: 3, name: "test4" },
  { id: 4, name: "test5" },
  { id: 5, name: "test6" },
  { id: 5, name: "test7" },
  { id: 6, name: "test8" }
];

const filteredArr = arr.filter(el => {
  const duplicate = seen.has(el.id);
  seen.add(el.id);
  return !duplicate;
});
Enter fullscreen mode Exit fullscreen mode

Discussion (7)

Collapse
larsonnn profile image
Lars Feldeisen
const filteredArr = arr.filter(
    (filterItem, index, self) =>
        index === self.findIndex((findItem) => findItem.id === filterItem.id),
);
Enter fullscreen mode Exit fullscreen mode
Collapse
lukeshiru profile image
LUKESHIRU

Consider that the Set approach is way faster.

Collapse
larsonnn profile image
Lars Feldeisen • Edited on

yes.
some more functions: LINK

in general a while loop is the fastest (but sometimes not, 98% of the time it will). its not recommended. Because filter is also using a while loop but also checks the values if undefined etc. It is safer to use filter. Or we need to check manually. Which would lead in a slower approach.

/* withLoop */
const found = [];
const newArray = [];

const withLoop = array => { 
  let i = 0;
  while(i < array.length) {
    if(!found.includes(array[i].id)) {
      found.push(array[i].id);
      newArray.push(array[i]);
    }
    ++i;
  }
  return newArray;
}
Enter fullscreen mode Exit fullscreen mode

The whole Set loop method it also fast, but when you need to return an array instead its slows the function down.

Thread Thread
peerreynders profile image
peerreynders • Edited on

in general a while loop is the fastest

Your sample code moves the temporary objects to the top level which exaggerates the difference.

but when you need to return an array instead its slows the function down.

There is no reason to store the objects in a Set in the first place so that conversion is entirely unnecessary.

One also has to keep in mind that V8 has a spread optimization which is going to penalize overuse of the spread syntax on other JavaScript engines.

Perflink

  1. withFilter
  2. withLoopSet
  3. withLoopIndexOf
  4. withLoop
  5. withSet
  6. withFor
  7. control (withFor)


Results for Chromium 96 on Ubuntu 20.04

The shape of that chart changes wildly with each run. To some degree that is to be expected as JavaScript performance is notoriously unpredictable.

Judging from the results on Firefox (94.0)

combining filter with Set doesn't help that much away from V8.

Thread Thread
lukeshiru profile image
LUKESHIRU

Yup, V8 does a lot of optimization with this kind of data, Quantum (I think that's the name of the current Firefox engine, the previous was Gecko if I remember correctly) is more in the "run as it is" kind of wagon. Is like V8 understands what you're trying to do with your filter/set and optimizes that internally to use "whiles", while Quantum just leaves that as it is, so you have to "pay" the price of filtering and checking a set in every iteration.

Thread Thread
larsonnn profile image
Lars Feldeisen • Edited on

There is no reason to store the objects in a Set in the first place so that conversion is entirely unnecessary.

Just to see what happens.

My bars on Firefox are so randomly but I guess it maybe has to do with my MacBook and the M1 Chip. And Firefox is not native here.

MacBook Pro (13-inch, M1, 2020) (m1 chip)

We should also mention Safari here.
Safari Version 15.1 (17612.2.9.1.20) (native)

Safari on m1

Safari data

Its not random you can reproduce it everytime. The performance is not even close to my Chrome Version 96.0.4664.55 (arm64)