How to Remove Duplicate Lines from Text (Online, Excel, and the Command Line)

#text #webdev #programming #tutorial

Duplicate lines creep into email lists, keyword exports, logs, and URL collections. Removing them by hand doesn't scale. A few settings decide whether the result is what you expect, so it helps to know them.

Decisions to make

Keep first or last occurrence: most tools keep the first and drop later repeats.
Preserve order or sort: deduplicating while keeping the original order is different from sorting the list and removing neighbors.
Case sensitivity: is Apple the same as apple? For emails and URLs, you usually want case-insensitive matching (with care: URL paths can be case-sensitive).
Whitespace: a trailing space makes two otherwise identical lines different. Trim before comparing.
Empty lines: decide whether to remove blank lines too.

In a browser tool

Paste your list, one item per line.
Choose case sensitivity and whether to trim whitespace.
Remove duplicates and check the count of lines removed.
Copy the result. Do this locally if the list contains personal data such as email addresses.

In the terminal

# Keep first occurrences, preserve order
awk '!seen[$0]++' input.txt > output.txt

# Sort and remove duplicates
sort -u input.txt > output.txt

# Case-insensitive unique after sorting
sort -f input.txt | uniq -i

Note that uniq only removes adjacent duplicates, so the input must be sorted first, which is why sort -u is the common shortcut.

In code

// JavaScript: preserves first occurrence order
const unique = [...new Set(lines)];

# Python
unique = list(dict.fromkeys(lines))

In a spreadsheet

Excel: Data, then Remove Duplicates, or use the UNIQUE function in current versions.
Google Sheets: the UNIQUE function returns the distinct rows.

Clean before you compare

Normalize the text first: trim spaces, lowercase where appropriate, and remove invisible characters. Most "why isn't this deduplicating" problems are hidden whitespace.

Frequently asked questions

How do I remove duplicate lines but keep the original order?

Use a tool or command that keeps the first occurrence, such as awk '!seen[$0]++' or a Set in code, rather than sorting.

Why does my list still show duplicates?

The lines likely differ by trailing spaces, capitalization, or invisible characters. Trim and normalize case first.

What is the difference between uniq and sort -u?

uniq removes only adjacent duplicates, so it needs sorted input. sort -u sorts and removes duplicates in one step.

Is it safe to paste a list into an online deduplicator?

Only if it runs in your browser without uploading. Avoid sending emails or other personal data to a server.

Try it: Remove Duplicate Lines — free, runs in your browser, nothing is uploaded.

Originally published at ilovekit.app.

DEV Community