JavaScript - remove duplicated lines

Hello there! 👋 😊

In this article, I would like to show you two ways to remove duplicated lines from some text in JavaScript. 🔠

Before we start, I would highly recommend you to check out runnable examples for the solution on our website:
JavaScript - remove duplicated lines

We're gonna use two methods:

• filter()
• reduce()

filter() based example

This approach uses a functional programming pattern.

On the text I've used set of operations to remove duplicated lines:

• split() method to split text into lines that take a newline character in 4 different variants which is a universal approach,
• filter() method that creates new array of elements tested with provided function,
• join() method to merge items back.

Practical example:

const newLineExpression = /\r\n|\n\r|\n|\r/g;

const removeDuplicatedLines = (text) => {
    return text.split(newLineExpression)
        .filter((item, index, array) => array.indexOf(item) === index)
        .join('\n');
};

// Usage example:

const text = `a
b
b
a
a
c
c`;

console.log(removeDuplicatedLines(text)); // a
                                          // b
                                          // c

You can run this example here

reduce() based example

This approach was created to show that it is possible to get the same effect as in the example above with reduce() method.

Practical example:

const newLineExpression = /\r\n|\n\r|\n|\r/g;

const removeDuplicatedLines = (text) => {
    const blocker = {}; // prevents lines dupplication
    return text.split(newLineExpression)
        .reduce((result, line) => {
            if (blocker.hasOwnProperty(line)) {
                return result;
            }
            blocker[line] = true;
            return result + line + '\n';
        }, '');
};

// Usage example:

const text = `a
b
b
a
a
c
c`;

console.log(removeDuplicatedLines(text)); // a
                                          // b
                                          // c

You can run this example here

If you found this solution useful you can react to this post or just leave a comment to let me know what you think. Thanks for reading and see you next time! 😊

---

Write to us! ✉

If you have any problem to solve or questions that no one can answer related to a React or JavaScript topic, or you're looking for a mentoring write to us on dirask.com -> Questions

Comments

[Adam Nathaniel Davis]

This is all you need:

const newLineExpression = /\r\n|\n\r|\n|\r/g;

const removeDuplicatedLines = text => [...new Set(text.split(newLineExpression))].join('\n');  

// Usage example:

const text = `a
b
b
a
a
c
c`;

console.log(removeDuplicatedLines(text)); // a
                                          // b
                                          // c

[Adam Nathaniel Davis]

In JavaScript, nearly any time that you are wanting to de-dupe an array (of any kind), the very first tool you reach for should be Set()

[Dirask-JavaScript]

Great solution! Thanks for the tips. 😊