DEV Community

Basic Javascript: Removing Duplicates from an Array

Matthew Shin on September 04, 2019

As someone who is currently looking for a software developer job, I know just how important it is to have a solid foundation when it comes to codin...

Read full post

Nick Taylor • Sep 4 '19 • Edited

With a Set, you can just do

function removeDuplicates(array) {
  return [...new Set(array)]
}

I have some other examples here.

Have a Handy JS Snippet You Want to Share?

Nick Taylor (he/him) ・ Jan 23 '18 ・ 2 min read

#javascript #showdev

lesha 🟨⬛️ • Sep 4 '19 • Edited

While I totally agree with you, this post is filed under #beginners. Beginner programmers need to build a solid understanding of how things work under the hood (doing deduplication manually) and then use abstractions like Set.

Alex K. • Sep 4 '19

While you do have a point there, I think array.splice(0, array.length, ...(new Set(array))) would be more confusing for beginners ;)

lesha 🟨⬛️ • Sep 4 '19

that's a nightmare tbh

Tails128 • Sep 4 '19

I would say that is something which should not be taught to new devs, cause it will push them in the wrong direction, to be fair.

Marc Schleeweiß • Sep 4 '19

Using a Set is literally part of the blog post. None of the other possible solutions from the post are low level. In fact they all use lambda expressions, which is even more complicated to grasp than a Set, which consists of unique values (per definition).

worc • Sep 4 '19

i don't know that i would call a set an abstraction of manual deduplication. for a beginner, a set just is a set of unique values. how you get there manually, or what implementation your runtime is using to guarantee that uniqueness doesn't feel like a beginner topic.

Ronan Connolly 🛠 • Mar 3 '20 • Edited

I get a TypeScript error for trying to expand the set.
This works though:

function removeDuplicates(array) {
  return Array.from(new Set(array));
}

Seems like the simplest solution.

Marc Schleeweiß • Sep 4 '19 • Edited

The only code snippet that works is the first one. All others throw some kind of error or have side effects.

forEach throws Uncaught SyntaxError: Unexpected token [ because of the weird function declaration.
map throws Uncaught SyntaxError: Unexpected token new. Why would anyone name a variable new?
Set does work, but modifies the original array. Nick Taylors solution [...new Set(array)] does the same but better.

I am not sure where you got these but I would suggest taking some time to at least test the snippets next time you do a blog post.

António Avelar • Sep 4 '19

that variable definition simply blows my mind... (let new;)

Rudoslav • Sep 4 '19

I was surprised that the author and others commenting did not actually realize this :)

But I guess it does not dissolve the point of the post.

James Thomson • Sep 4 '19

Don't forget reduce :)

const items = ['a','b','c','a','a','d','b','d','d','c'];

function removeDuplicates (arr){
  return arr.reduce((acc, curr) => acc.includes(curr) ? acc : [...acc, curr], []);
}

Daniel Hillmann • Sep 4 '19

I was surprised this was not included :-)

jrdleto • Sep 5 '19

Can't figure out, what does [], in itterable condition

James Thomson • Sep 6 '19

The reduce function uses and accumulator as part of its logic. The [] is the starting value (an empty array) of the accumulator.

Simplified, using the reduce function to produce a total based on an array of values, we could start the reduce loop with the value 0.

const arr = [1,2,3,4,5,6,7,8];

const newArr = arr.reduce((acc, cur) => acc + cur, 0) // 36

We could also start it with another value, perhaps as a running total.

More on reduce: developer.mozilla.org/en-US/docs/W...

Curtis Fenner • Sep 4 '19 • Edited

You generally don't need forEach, a regular old for loop looks nicer. You also don't need the if; it's not harmful to write the same key twice.

for (let x of array) {
    x[v] = true;
}

However, it's important to highlight the differences between these implementations. This one, for example, only works when your array contains only strings.

I really can't condone using map as forEach. A more interesting variant would use spreading with reduce, like

array.reduce((x, v) => ({[v]: true, ...x}), {})

but again this has the limitation that it only works with strings.

JonasNo • Jan 28 '20

How do you implement "array.reduce((x, v) => ({[v]: true, ...x}), {})" in a removeDuplicates function? Im unfamiliar with that syntax.

Im running performance tests and so far this function is the fastest one:
function removeDuplicates3(array) {
array.splice(0, array.length, ...(new Set(array)))
}
but I want to test your suggestion too.

Verdotte • Sep 4 '19

This implementation maybe helpful for beginners

function removeDuplicates(array){
 let new = []
 for(let val of array) {
  if (!isExist(new, val)) 
  new.push(val)
 }
 return new
}

function isExist (arr, val){
 for (let i of arr){
  if (i == val) return true
 }
 return false
}

Matt Ellen-Tsivintzeli • Sep 4 '19 • Edited

While your map example works (apart from the variable name new), I would say that's really a reduce solution.

arr.reduce((acc, item) => 
                       { 
                         if(!acc.includes(item)) 
                         { 
                           acc.push(item); 
                         } 
                         return acc;
                       }, []);

I wouldn't use map for this, as it's not a straight, per item transform.

Took me a minute to figure out your fitler example :D

Venkatesh KL • Sep 5 '19

All the best with your learning @mshin1995 . Nice that you are looking for different approaches to solve the same problem, that's a pretty good way of learning good practices. I am saying this because, often I overlook a problem once it is solved. So, it is a very appreciable thing to solve it different ways.

All of that apart, I think Array's map function has a specific purpose. It is supposed to be used when we want to create a new array based on each value of an existing array. So, it may not be semantically good way to solve this. The other ones with the for loop, forEach and reduce are much better approaches.

I would say that once we understand the reasoning behind each utility that is provided by the js engines, we would actually find their purpose much clear. This won't help anyone initially, but in long run and to maintain long running processes, these things have a great impact.

All the best. Cheers :)

JonasNo • Jan 28 '20

Performance test shows this function is the fastest one:
function removeDuplicates3(array) {
array.splice(0, array.length, ...(new Set(array)))
}

Setup js, executed for each test:
function getRandomInt(max) {
return Math.floor(Math.random() * Math.floor(max));
}
const base = ["a", "b", "c", "d", "e", "f", "g", "h"];
const array_test_length = 5000;
var arr = [];
arr.length = array_test_length;
for (let i = 0, len = arr.length, blen = base.length; i < len; ++i) {
arr[i] = base[getRandomInt(blen)];
}

function removeDuplicates1(array) {
return array.filter((a, b) => array.indexOf(a) === b)
}

function removeDuplicates2(array) {
let x = {};
array.forEach(function(i) {
if(!x[i]) {
x[i] = true
}
})
return Object.keys(x)
}

function removeDuplicates3(array) {
array.splice(0, array.length, ...(new Set(array)))
}

function removeDuplicates4(array) {
let a = []
array.map(x => {
if(!a.includes(x)) {
a.push(x)
}
})
return a
}

function removeDuplicates5(array) {
return [...new Set(array)]
}

function removeDuplicates6(arr){
return arr.reduce((acc, curr) => acc.includes(curr) ? acc : [...acc, curr], []);
}

function removeDuplicates7(array){
let a = []
for(let val of array) {
if (!isExist(a, val))
a.push(val)
}
return a
}
function isExist (arr, val){
for (let i of arr){
if (i == val) return true
}
return false
}

function removeDuplicates8(array){
return array.reduce((acc, item) => {
if(!acc.includes(item)) {
acc.push(item);
}
return acc;
}, []);
}

Each test goes like this:
var new_arr = removeDuplicates1(arr); // only need to change the number

Angel Claudio • Sep 4 '19 • Edited

Hi Matthew thanks for your contribution! For your second example, you have 2 typos, brackets instead of parenthesis next to function, and Objects instead of Object. Here it is corrected:

function removeDuplicates(array) {
let x = {};
array.forEach(function(i) {
if (!x[i]) {
x[i] = true;
}
});
return Object.keys(x);
}