Cleaning messy lists in the browser: removing duplicate lines

Duplicate lines are everywhere: a list merged from two exports, an order report that repeats the order ID on every line item, or a log file that prints the same error again and again. Removing them by hand is slow, and the duplicates that differ by one capital letter or a trailing space are the easiest to miss.

I built a small browser tool that removes duplicate lines without uploading anything, and this post walks through the logic behind it.

Keep the first occurrence, drop the rest

The core of a duplicate remover is a Set. Walk the lines in order, keep a record of what you have seen, and skip anything that repeats:

function removeDuplicateLines(text, { caseInsensitive = false } = {}) {
  const lines = text.split(/\r?\n/);
  const seen = new Set();
  const result = [];

  for (const line of lines) {
    const key = caseInsensitive ? line.trim().toLowerCase() : line;
    if (!seen.has(key)) {
      seen.add(key);
      result.push(line);
    }
  }

  return result.join("\n");
}

Two details matter more than they look:

• Preserve order. Sorting the output is a separate feature. Most people want their original list back, minus the repeats.
• Case sensitivity is a choice. To JavaScript, Apple and apple are different strings. For a name list you may want them merged; for product codes or usernames you probably do not. Expose it as an option instead of guessing.

That is the whole algorithm. Everything else is interface.

Why not just sort -u?

If you live in a terminal, this does the job:

sort -u input.txt > output.txt

But it sorts, it only matches exact strings, and a single trailing space silently keeps a duplicate alive. A file that "didn't deduplicate" is almost always a whitespace problem, not a logic problem.

The browser tool exists for the cases where you want to see the before and after, and choose the comparison mode without writing a one-off script.

The near-duplicate trap

The duplicates that cause trouble are often not identical:

Pair	Why it survives an exact match
Apple / apple	Different case
apple / apple	Trailing space
New York / New York	Extra whitespace
[e-mail] / [e-mail]	Mixed case in an email

If your list came from a spreadsheet, a form, or several people typing, expect near-duplicates. Trim spaces and normalise case first, then deduplicate. Otherwise the tool does exactly what you asked and removes only the lines that are truly identical.

A worked example

Input:

apples
bananas
Apples
bananas
cherries

With exact matching and preserve order, the output is:

apples
bananas
Apples
cherries

Apples survives because the capital A makes it a different line. bananas appears once. Turn on case-insensitive mode and the output becomes:

apples
bananas
cherries

The first spelling wins, which is why checking inconsistent capitalisation before you delete is worth the extra minute.

When you should not remove duplicates

Removing duplicates is not always correct. Keep them when the repetition carries information:

• Order and invoice lines — the same product can legitimately appear twice; the quantity matters.
• Logs and audit trails — a repeated error at 10:00 and 11:00 is a pattern, not noise.
• Votes, reactions, and counts — one entry per event is the data.
• Survey answers — two people can give the same answer and both count.

A simple test: if deleting the second copy loses a fact, do not delete it.

Try it

• Duplicate line remover (runs locally in the browser): https://yourutilityhub.com/text/duplicate-line-remover
• Extra spaces remover: https://yourutilityhub.com/text/remove-extra-spaces
• Full walkthrough with spreadsheet and command-line methods: https://yourutilityhub.com/blog/remove-duplicate-lines

No signup, and nothing is uploaded — the text is processed in the page itself. If you clean lists regularly, the command line still wins for batch jobs. For a one-off paste, this is faster.

Comments

[KomoPDF]

Handling duplicate removal directly in browser memory without sending data to a remote server is the cleanest pattern for user privacy.

When dealing with large text exports or mixed formats, two minor edge cases often catch users off guard:

1. Unicode normalization: Characters like accented letters or non-breaking spaces (\u00A0 vs \u0020) can look identical on screen but produce different hashes in a JavaScript Set. Running line.normalize('NFC') alongside trim() helps eliminate hidden character mismatches.
2. Large paste payloads: For files exceeding tens of thousands of lines, running the split and Set evaluation inside a Web Worker prevents the browser main thread from stuttering or triggering an unresponsive script alert.

Solid breakdown of the trade-offs between exact terminal sorting and browser-based list cleaning.