DEV Community

Cover image for Basic Javascript: Removing Duplicates from an Array
Matthew Shin
Matthew Shin

Posted on

Basic Javascript: Removing Duplicates from an Array

As someone who is currently looking for a software developer job, I know just how important it is to have a solid foundation when it comes to coding. Although practicing for coding challenges that test these fundamentals might not be the most exciting, they are definitely practical by making your brain think in different ways. I thought that it would be helpful for me personally and for anyone reviewing for those technical interviews to go back to the basics and touch on some core topics. For this week, I wanted to come up with multiple ways to remove duplicates within an array.

1. filter()

function removeDuplicates(array) {
  return array.filter((a, b) => array.indexOf(a) === b)
};
Enter fullscreen mode Exit fullscreen mode

2. forEach()

function removeDuplicates(array) {
  let x = {};
  array.forEach(function(i) {
    if(!x[i]) {
      x[i] = true
    }
  })
  return Object.keys(x)
};
Enter fullscreen mode Exit fullscreen mode

Checks to see if each element in the array is unique.

3. Set

function removeDuplicates(array) {
  array.splice(0, array.length, ...(new Set(array)))
};
Enter fullscreen mode Exit fullscreen mode

Creating a new Set stores unique elements from the array. The easiest way in my opinion

4. map()

function removeDuplicates(array) {
  let a = []
  array.map(x => 
    if(!a.includes(x) {
      a.push(x)
    })
  return a
};
Enter fullscreen mode Exit fullscreen mode

Maps through the array and adds unique elements to a new array.

I hope some people found this helpful! What are some other solutions that you guys can think of?

Discussion (27)

Collapse
nickytonline profile image
Nick Taylor (he/him) • Edited

With a Set, you can just do

function removeDuplicates(array) {
  return [...new Set(array)]
}
Enter fullscreen mode Exit fullscreen mode

I have some other examples here.

Collapse
lesha profile image
lesha 🟨⬛️ • Edited

While I totally agree with you, this post is filed under #beginners. Beginner programmers need to build a solid understanding of how things work under the hood (doing deduplication manually) and then use abstractions like Set.

Collapse
clarity89 profile image
Alex K.

While you do have a point there, I think array.splice(0, array.length, ...(new Set(array))) would be more confusing for beginners ;)

Thread Thread
lesha profile image
lesha 🟨⬛️

that's a nightmare tbh

Thread Thread
tails128 profile image
Tails128

I would say that is something which should not be taught to new devs, cause it will push them in the wrong direction, to be fair.

Collapse
mschleeweiss profile image
Marc Schleeweiß

Using a Set is literally part of the blog post. None of the other possible solutions from the post are low level. In fact they all use lambda expressions, which is even more complicated to grasp than a Set, which consists of unique values (per definition).

Collapse
worc profile image
worc

i don't know that i would call a set an abstraction of manual deduplication. for a beginner, a set just is a set of unique values. how you get there manually, or what implementation your runtime is using to guarantee that uniqueness doesn't feel like a beginner topic.

Collapse
ronancodes profile image
Ronan Connolly 🛠 • Edited

I get a TypeScript error for trying to expand the set.
This works though:

function removeDuplicates(array) {
  return Array.from(new Set(array));
}

Seems like the simplest solution.

Collapse
mschleeweiss profile image
Marc Schleeweiß • Edited

The only code snippet that works is the first one. All others throw some kind of error or have side effects.

  1. forEach throws Uncaught SyntaxError: Unexpected token [ because of the weird function declaration.

  2. map throws Uncaught SyntaxError: Unexpected token new. Why would anyone name a variable new?

  3. Set does work, but modifies the original array. Nick Taylors solution [...new Set(array)] does the same but better.

I am not sure where you got these but I would suggest taking some time to at least test the snippets next time you do a blog post.

Collapse
antonioavelar profile image
António Avelar

that variable definition simply blows my mind... (let new;)

Collapse
rudoslav profile image
Rudoslav

I was surprised that the author and others commenting did not actually realize this :)

But I guess it does not dissolve the point of the post.

Collapse
jamesthomson profile image
James Thomson

Don't forget reduce :)

const items = ['a','b','c','a','a','d','b','d','d','c'];

function removeDuplicates (arr){
  return arr.reduce((acc, curr) => acc.includes(curr) ? acc : [...acc, curr], []);
}
Collapse
hilleer profile image
Daniel Hillmann

I was surprised this was not included :-)

Collapse
jrdleto profile image
jrdleto

Can't figure out, what does [], in itterable condition

Collapse
jamesthomson profile image
James Thomson

The reduce function uses and accumulator as part of its logic. The [] is the starting value (an empty array) of the accumulator.

Simplified, using the reduce function to produce a total based on an array of values, we could start the reduce loop with the value 0.

const arr = [1,2,3,4,5,6,7,8];

const newArr = arr.reduce((acc, cur) => acc + cur, 0) // 36

We could also start it with another value, perhaps as a running total.

More on reduce: developer.mozilla.org/en-US/docs/W...

Collapse
curtisfenner profile image
Curtis Fenner • Edited

You generally don't need forEach, a regular old for loop looks nicer. You also don't need the if; it's not harmful to write the same key twice.

for (let x of array) {
    x[v] = true;
}

However, it's important to highlight the differences between these implementations. This one, for example, only works when your array contains only strings.

I really can't condone using map as forEach. A more interesting variant would use spreading with reduce, like

array.reduce((x, v) => ({[v]: true, ...x}), {})

but again this has the limitation that it only works with strings.

Collapse
jonasno profile image
JonasNo

How do you implement "array.reduce((x, v) => ({[v]: true, ...x}), {})" in a removeDuplicates function? Im unfamiliar with that syntax.

Im running performance tests and so far this function is the fastest one:
function removeDuplicates3(array) {
array.splice(0, array.length, ...(new Set(array)))
}
but I want to test your suggestion too.

Collapse
verdotte1 profile image
Verdotte

This implementation maybe helpful for beginners

function removeDuplicates(array){
 let new = []
 for(let val of array) {
  if (!isExist(new, val)) 
  new.push(val)
 }
 return new
}

function isExist (arr, val){
 for (let i of arr){
  if (i == val) return true
 }
 return false
}
Collapse
mellen profile image
Matt Ellen • Edited

While your map example works (apart from the variable name new), I would say that's really a reduce solution.

arr.reduce((acc, item) => 
                       { 
                         if(!acc.includes(item)) 
                         { 
                           acc.push(item); 
                         } 
                         return acc;
                       }, []);

I wouldn't use map for this, as it's not a straight, per item transform.

Took me a minute to figure out your fitler example :D

Collapse
klvenky profile image
Venkatesh KL

All the best with your learning @mshin1995 . Nice that you are looking for different approaches to solve the same problem, that's a pretty good way of learning good practices. I am saying this because, often I overlook a problem once it is solved. So, it is a very appreciable thing to solve it different ways.

All of that apart, I think Array's map function has a specific purpose. It is supposed to be used when we want to create a new array based on each value of an existing array. So, it may not be semantically good way to solve this. The other ones with the for loop, forEach and reduce are much better approaches.

I would say that once we understand the reasoning behind each utility that is provided by the js engines, we would actually find their purpose much clear. This won't help anyone initially, but in long run and to maintain long running processes, these things have a great impact.

All the best. Cheers :)

Collapse
jonasno profile image
JonasNo

Performance test shows this function is the fastest one:
function removeDuplicates3(array) {
array.splice(0, array.length, ...(new Set(array)))
}

Setup js, executed for each test:
function getRandomInt(max) {
return Math.floor(Math.random() * Math.floor(max));
}
const base = ["a", "b", "c", "d", "e", "f", "g", "h"];
const array_test_length = 5000;
var arr = [];
arr.length = array_test_length;
for (let i = 0, len = arr.length, blen = base.length; i < len; ++i) {
arr[i] = base[getRandomInt(blen)];
}

function removeDuplicates1(array) {
return array.filter((a, b) => array.indexOf(a) === b)
}

function removeDuplicates2(array) {
let x = {};
array.forEach(function(i) {
if(!x[i]) {
x[i] = true
}
})
return Object.keys(x)
}

function removeDuplicates3(array) {
array.splice(0, array.length, ...(new Set(array)))
}

function removeDuplicates4(array) {
let a = []
array.map(x => {
if(!a.includes(x)) {
a.push(x)
}
})
return a
}

function removeDuplicates5(array) {
return [...new Set(array)]
}

function removeDuplicates6(arr){
return arr.reduce((acc, curr) => acc.includes(curr) ? acc : [...acc, curr], []);
}

function removeDuplicates7(array){
let a = []
for(let val of array) {
if (!isExist(a, val))
a.push(val)
}
return a
}
function isExist (arr, val){
for (let i of arr){
if (i == val) return true
}
return false
}

function removeDuplicates8(array){
return array.reduce((acc, item) => {
if(!acc.includes(item)) {
acc.push(item);
}
return acc;
}, []);
}

Each test goes like this:
var new_arr = removeDuplicates1(arr); // only need to change the number

Collapse
angelclaudio profile image
Angel Claudio • Edited

Hi Matthew thanks for your contribution! For your second example, you have 2 typos, brackets instead of parenthesis next to function, and Objects instead of Object. Here it is corrected:

function removeDuplicates(array) {
let x = {};
array.forEach(function(i) {
if (!x[i]) {
x[i] = true;
}
});
return Object.keys(x);
}

Collapse
jurgentreep profile image
Jurgen Treep

Making it more complicated than it has to be is bad practice in my opinion. Just keep it simple.

Collapse
cmkr profile image
Chintha Reddy

Can any one explain How filter Method Works Internally

Collapse
ruslangonzalez profile image
Ruslán González

Very useful will test it. Thanks for sharing.

Collapse
ham8821 profile image
Emma Ham

Thanks a lot! will definitely try👍

Collapse
vssenko profile image
Vitaly Senko

Of course only options #2 and #3 are viable.