Basic Javascript: Removing Duplicates from an Array

As someone who is currently looking for a software developer job, I know just how important it is to have a solid foundation when it comes to coding. Although practicing for coding challenges that test these fundamentals might not be the most exciting, they are definitely practical by making your brain think in different ways. I thought that it would be helpful for me personally and for anyone reviewing for those technical interviews to go back to the basics and touch on some core topics. For this week, I wanted to come up with multiple ways to remove duplicates within an array.

1. filter()

function removeDuplicates(array) {
  return array.filter((a, b) => array.indexOf(a) === b)
};

2. forEach()

function removeDuplicates(array) {
  let x = {};
  array.forEach(function(i) {
    if(!x[i]) {
      x[i] = true
    }
  })
  return Object.keys(x)
};

Checks to see if each element in the array is unique.

3. Set

function removeDuplicates(array) {
  array.splice(0, array.length, ...(new Set(array)))
};

Creating a new Set stores unique elements from the array. The easiest way in my opinion

4. map()

function removeDuplicates(array) {
  let a = []
  array.map(x => 
    if(!a.includes(x) {
      a.push(x)
    })
  return a
};

Maps through the array and adds unique elements to a new array.

I hope some people found this helpful! What are some other solutions that you guys can think of?

Comments

[Nick Taylor]

With a Set, you can just do

function removeDuplicates(array) {
  return [...new Set(array)]
}

I have some other examples here.

Have a Handy JS Snippet You Want to Share?

Nick Taylor (he/him) ・ Jan 23 '18 ・ 2 min read

#javascript #showdev

[lesha 🟨⬛️]

While I totally agree with you, this post is filed under #beginners. Beginner programmers need to build a solid understanding of how things work under the hood (doing deduplication manually) and then use abstractions like Set.

[Alex K.]

While you do have a point there, I think array.splice(0, array.length, ...(new Set(array))) would be more confusing for beginners ;)

[lesha 🟨⬛️]

that's a nightmare tbh

[Tails128]

I would say that is something which should not be taught to new devs, cause it will push them in the wrong direction, to be fair.

[Marc Schleeweiß]

Using a Set is literally part of the blog post. None of the other possible solutions from the post are low level. In fact they all use lambda expressions, which is even more complicated to grasp than a Set, which consists of unique values (per definition).

[worc]

i don't know that i would call a set an abstraction of manual deduplication. for a beginner, a set just is a set of unique values. how you get there manually, or what implementation your runtime is using to guarantee that uniqueness doesn't feel like a beginner topic.

[Ronan Connolly 🛠]

I get a TypeScript error for trying to expand the set.
This works though:

function removeDuplicates(array) {
  return Array.from(new Set(array));
}

Seems like the simplest solution.

[Marc Schleeweiß]

The only code snippet that works is the first one. All others throw some kind of error or have side effects.

1. forEach throws Uncaught SyntaxError: Unexpected token [ because of the weird function declaration.

2. map throws Uncaught SyntaxError: Unexpected token new. Why would anyone name a variable new?

3. Set does work, but modifies the original array. Nick Taylors solution [...new Set(array)] does the same but better.

I am not sure where you got these but I would suggest taking some time to at least test the snippets next time you do a blog post.

[António Avelar]

that variable definition simply blows my mind... (let new;)

[Rudoslav]

I was surprised that the author and others commenting did not actually realize this :)

But I guess it does not dissolve the point of the post.

[James Thomson]

Don't forget reduce :)

const items = ['a','b','c','a','a','d','b','d','d','c'];

function removeDuplicates (arr){
  return arr.reduce((acc, curr) => acc.includes(curr) ? acc : [...acc, curr], []);
}

[Daniel Hillmann]

I was surprised this was not included :-)

[jrdleto]

Can't figure out, what does [], in itterable condition

[James Thomson]

The reduce function uses and accumulator as part of its logic. The [] is the starting value (an empty array) of the accumulator.

Simplified, using the reduce function to produce a total based on an array of values, we could start the reduce loop with the value 0.

const arr = [1,2,3,4,5,6,7,8];

const newArr = arr.reduce((acc, cur) => acc + cur, 0) // 36

We could also start it with another value, perhaps as a running total.

More on reduce: developer.mozilla.org/en-US/docs/W...

[Curtis Fenner]

You generally don't need forEach, a regular old for loop looks nicer. You also don't need the if; it's not harmful to write the same key twice.

for (let x of array) {
    x[v] = true;
}

However, it's important to highlight the differences between these implementations. This one, for example, only works when your array contains only strings.

I really can't condone using map as forEach. A more interesting variant would use spreading with reduce, like

array.reduce((x, v) => ({[v]: true, ...x}), {})

but again this has the limitation that it only works with strings.

[JonasNo]

How do you implement "array.reduce((x, v) => ({[v]: true, ...x}), {})" in a removeDuplicates function? Im unfamiliar with that syntax.

Im running performance tests and so far this function is the fastest one:
function removeDuplicates3(array) {
array.splice(0, array.length, ...(new Set(array)))
}
but I want to test your suggestion too.

[Verdotte]

This implementation maybe helpful for beginners

function removeDuplicates(array){
 let new = []
 for(let val of array) {
  if (!isExist(new, val)) 
  new.push(val)
 }
 return new
}

function isExist (arr, val){
 for (let i of arr){
  if (i == val) return true
 }
 return false
}

[Matt Ellen-Tsivintzeli]

While your map example works (apart from the variable name new), I would say that's really a reduce solution.

arr.reduce((acc, item) => 
                       { 
                         if(!acc.includes(item)) 
                         { 
                           acc.push(item); 
                         } 
                         return acc;
                       }, []);

I wouldn't use map for this, as it's not a straight, per item transform.

Took me a minute to figure out your fitler example :D

[Venkatesh KL]

All the best with your learning @mshin1995 . Nice that you are looking for different approaches to solve the same problem, that's a pretty good way of learning good practices. I am saying this because, often I overlook a problem once it is solved. So, it is a very appreciable thing to solve it different ways.

All of that apart, I think Array's map function has a specific purpose. It is supposed to be used when we want to create a new array based on each value of an existing array. So, it may not be semantically good way to solve this. The other ones with the for loop, forEach and reduce are much better approaches.

I would say that once we understand the reasoning behind each utility that is provided by the js engines, we would actually find their purpose much clear. This won't help anyone initially, but in long run and to maintain long running processes, these things have a great impact.

All the best. Cheers :)

[JonasNo]

Performance test shows this function is the fastest one:
function removeDuplicates3(array) {
array.splice(0, array.length, ...(new Set(array)))
}

Setup js, executed for each test:
function getRandomInt(max) {
return Math.floor(Math.random() * Math.floor(max));
}
const base = ["a", "b", "c", "d", "e", "f", "g", "h"];
const array_test_length = 5000;
var arr = [];
arr.length = array_test_length;
for (let i = 0, len = arr.length, blen = base.length; i < len; ++i) {
arr[i] = base[getRandomInt(blen)];
}

function removeDuplicates1(array) {
return array.filter((a, b) => array.indexOf(a) === b)
}

function removeDuplicates2(array) {
let x = {};
array.forEach(function(i) {
if(!x[i]) {
x[i] = true
}
})
return Object.keys(x)
}

function removeDuplicates3(array) {
array.splice(0, array.length, ...(new Set(array)))
}

function removeDuplicates4(array) {
let a = []
array.map(x => {
if(!a.includes(x)) {
a.push(x)
}
})
return a
}

function removeDuplicates5(array) {
return [...new Set(array)]
}

function removeDuplicates6(arr){
return arr.reduce((acc, curr) => acc.includes(curr) ? acc : [...acc, curr], []);
}

function removeDuplicates7(array){
let a = []
for(let val of array) {
if (!isExist(a, val))
a.push(val)
}
return a
}
function isExist (arr, val){
for (let i of arr){
if (i == val) return true
}
return false
}

function removeDuplicates8(array){
return array.reduce((acc, item) => {
if(!acc.includes(item)) {
acc.push(item);
}
return acc;
}, []);
}

Each test goes like this:
var new_arr = removeDuplicates1(arr); // only need to change the number

[Angel Claudio]

Hi Matthew thanks for your contribution! For your second example, you have 2 typos, brackets instead of parenthesis next to function, and Objects instead of Object. Here it is corrected:

function removeDuplicates(array) {
let x = {};
array.forEach(function(i) {
if (!x[i]) {
x[i] = true;
}
});
return Object.keys(x);
}

[Jurgen Treep]

Making it more complicated than it has to be is bad practice in my opinion. Just keep it simple.