DEV Community

JavaScript: How to Remove Duplicate Values from Arrays

Will Harris on July 03, 2020

Originally posted on Will's blog In a previous post we saw how to determine whether a JavaScript array contains duplicate values. Today, I want...

Read full post

Mike Talbot ⭐ • Jul 3 '20 • Edited

Surely Set must be faster - the whole thing is an O(N) operation with Set, but it's getting towards O(N**2) with anything that has to scan like includes or filter... (The reduce version is better than O(N**2) as it is only having to test the currently created unique list)

I decided to see just how different it is. In this example on jsperf, you can see that the "reduce" is around 180x slower and the indexOf even worse at almost 380x slower. Also the version of "reduce" I've used is slightly faster than your one shown, because it doesn't make an array on each loop and just modifies the one being created.

Will Harris • Jul 3 '20

Thanks for throwing together that jsperf test Mike!

Set does appear to be much faster, which is fantastic because the syntax is so much easier to grok (for me, at least).

Mike Talbot ⭐ • Jul 3 '20

Yes, nice that good syntax has great performance haha! Not a normal happy thing that I find! (Like the immutability is a nicer syntax but comes with a penalty).

Evan Kennedy • Jul 3 '20 • Edited

The approach below is O(n) and should work without ES6. The issue with some of the approaches above is they are using Array.indexOf or Array.includes, which is basically another loop through the array. You basically take a performance hit when the searched values are located near the end of the array. The only downside to the method below is that it's memory intensive for large arrays. You're essentially giving up memory for performance.

const originalArray = [1, 2, 3, 4, 1, 2, 3, 4];

let key = {};

const uniqueArray = originalArray.filter((item) => {
  if (key[item]) return false;

  key[item] = true;
  return true;
});

willsmart • Jul 3 '20 • Edited

A similar function using Set is

function withoutDuplicates(inputArray) {
  const trackerSet=new Set();
  return inputArray.filter(
    v => !trackerSet.has(v) && trackerSet.add(v)
  ) // add returns the Set instance, which is truthy. You may wish to add a more explicit truthy return
}

These have the main advantage over Array.from(new Set(inputArray)) in that there's no need to iterate the Set after filling it with values, but I've done a little profiling and my snippet has pretty much the same characteristics and timing as Array.from(new Set(inputArray)) (just a little bit slower), even for very large arrays.

tbh, I think the big win is that they aren't leaning on JS Set's latent insertion-order stability, making the code more WYSIWYG, and more portable since in many other C-style languages Sets don't have that guarantee.

Will Harris • Jul 4 '20

Thanks Evan! That's also a great way to approach the problem.

James McConnon • Jul 4 '20

Awesome article. First i am seeing sets. Also glad to read comment below that its runtime is better than expected because to me it is more readable. Also i think allot of the array prototype functions can make the code harder to read in the long run. Assuming you know a bit of background on sets, that last example is so concise!

That being said, could you explain how the ... spread operator works with it? Trying to get my head around that at the moment.

Will Harris • Jul 4 '20

Thanks James!

Because the Set object implements the iterator property, we can use the ES6 spread syntax to cast the values from the Set into an array.

It really is doing the same thing as the second part of that example, where I called Array.from().

James McConnon • Jul 5 '20

Thankyou that makes sense. Spread operator has been a tough one to learn for me. I always feel like i kind if have an idea of what it is doing in one situation but struggle to use it myself.

SavagePixie • Jul 4 '20

Sets don't come with the same prototype functions as arrays. They have a forEach method, but no map or filter, for example (I think). The spread operator and square braces there are simply extracting the values from the set and turning them into an array.

Felipe Pontes • Jul 4 '20

Set works great when you are dealing with primitive values. But if you are working with some list of items (e.g. list of comment authors in this post), you're better off using a key mapping approach:

const mapping = {}
const unique = array.filter((item) => {
  if (item.id in mapping) return false
  mapping[item.id] = true
  return true
})

Michael R. • Oct 27 '20

Is there any way to count how many times the duplicate object(s) appears in the array before eliminating said duplicates using this method? Asking for a noob trying to solve an annoying kata. :)

Reference link

miggu • Jul 7 '20

What is quadratic time?

Kyle Stephens • Jul 4 '20

Use Set:

developer.mozilla.org/en-US/docs/W...

jesuscovam • Jul 3 '20

too much coolness in one function

shahid • Jul 7 '20

stackblitz.com/edit/typescript-5q3ktr
O (N) time

caoshouse • Jul 4 '20 • Edited

let originalArray = [1, 2, 3, 4, 1, 2, 3, 4];
for(var i=0,obj={};i<originalArray.length;i++){
obj[originalArray[i]]=null
}
let noDuplicates=Object.keys(obj)

Mark Voorberg • Jul 4 '20 • Edited

Same using .reduce()...

let myArray = [1,2,2,"hello world",4,5,5, {moo: "cow"}];
let myObject = myArray.reduce((acc, x) => {acc[x]=0; return acc}, {});
let noDuplicates = Object.keys(myObject);