Remove Duplicate Lines in the Browser: Case, Empty Lines and Which Copy You Keep

I work on OnePassApps. We build browser studios that do their work in the tab instead of uploading your files, and Text Studio is the one for plain text: eight tools on one tab bar - Clean, Case, Duplicate lines, Sort lines, Find & replace, Diff, Markdown and Base64.

The Duplicate lines tab is the one that looks trivial until you ask what "duplicate" means. Is Apple the same line as apple? Is " apple" (with a leading space) the same as apple? When a line repeats three times, which copy survives, and where does it end up in the output? This is about those rules rather than the button: the three options, and what the tool actually does with case, empty lines and repeated lines.

The page describes itself in one line: "Count, clean, compare and convert text locally. Your content stays in this browser tab."

What the tab gives you

One panel, three steps. 01 Input text - a box with a live counter (2,000,000 characters on this tab) and a live stats row: Words, Characters, Sentences, Lines and UTF-8 bytes. 02 Options - three controls: Keep duplicate order, Case-sensitive and Keep every empty line. 03 Result - "Unique lines shows on the stage, ready to copy, download or use as the new input."

The action button reads "Remove duplicate lines" and becomes "Remove again" once a run has happened. The result panel then reports the size of what it produced and how long it took ("Duplicates removed in 0.1 s"), and offers Use as input, Download .txt and Copy. The stage footer reads "Text transformed locally." and the panel says "Nothing leaves this tab."

You can drop a file in instead of pasting: the drop area takes ".txt, .md, .csv and other text up to 8 MiB", and the tab says "Processing stays in this browser." The file input itself accepts .txt,.text,.md,.markdown,.csv,.tsv,.json,.log,.xml,.yaml,.yml,.html,.srt,.vtt, plus the text/plain, text/markdown and text/csv types.

The three options

Keep duplicate order is a dropdown with two entries: "First occurrence" and "Last occurrence". It ships on First occurrence.

Case-sensitive ships on (ticked), and Keep every empty line ships on (ticked) as well. Those two defaults are the whole reason the tab is worth a look: a default-on case-sensitive match will happily keep Apple and apple as two different lines, and a default-on empty-line setting keeps your blank lines.

Run 1: the defaults

Input, eight lines including one empty line:

apple
banana
apple
Apple
 apple
banana

apple

With the defaults (First occurrence, Case-sensitive on, Keep every empty line on), the result stage reported "Duplicates removed in 0.1 s" and "26 characters · 5 lines". The output was:

apple
banana
Apple
 apple

Read that carefully, because every line in it is a decision:

• apple appears three times in the input and survives once. The first occurrence won, so the output keeps the lowercase spelling, not the capitalised one that came later.
• Apple survives as its own line. Case-sensitive is on, so it is a different line from apple.
• apple (with a leading space) survives as its own line too. The comparison is on the exact line, so a leading space makes it different.
• banana appears twice and survives once.
• The empty line survives, because Keep every empty line is on.

Run 2: case-insensitive

Untick Case-sensitive and run the same input again. The result is "20 characters · 4 lines":

apple
banana
 apple

Apple is now folded into apple, so the output has one fewer line. Note the leading-space line still stands on its own: the tool compares the line as it is, it does not trim whitespace before comparing. So "case-insensitive" here means exactly that - it changes the letter comparison and nothing else.

Run 3: and the empty line goes too

Untick Keep every empty line as well and the same input gives "19 characters · 3 lines":

apple
banana
 apple

Three lines now. The empty line is gone, the case difference is gone, and the whitespace difference is still there. Those are three separate switches doing three separate jobs, which is worth knowing before you assume "de-duplicate" means one particular thing.

Run 4: which copy survives, and where does it land

Keep duplicate order decides both. With the input a, b, a, c:

• First occurrence gives a, b, c - the surviving copy keeps the first position the line ever had.
• Last occurrence gives b, a, c - the surviving copy moves to the last position that line appeared in, and the other lines shift up to close the gap.

Both runs report the same size, "5 characters · 3 lines". The set of unique lines is identical; only the order differs. That is the whole difference between the two dropdown entries.

The same thing with a line that repeats three times, input keep, x, keep, y, keep, under Last occurrence, gives x, y, keep: the surviving copy lands at the end, where the third occurrence was.

The output file

The result panel offers "Download .txt", and that button really writes a file. Reading the blob it creates back gives a text/plain;charset=utf-8 payload of exactly 26 bytes for the first run:

apple
banana
Apple
 apple

26 bytes, matching the "26 characters · 5 lines" the panel reported. The trailing newline on the last line is in the file, and the empty line from the input is not in it as an empty line - it is the newline that separates apple from the end of the file, which is why the panel counts 5 lines while the text shows 4 visible ones.

What the tab does not do

It does not sort. The output comes out in the order the surviving lines were in the input, and the Sort lines tab is the separate tool for reordering. It does not trim or normalise whitespace, and it does not rewrite the lines it keeps - the output is made of the exact strings that were in the input.

It also does not touch the original. "Use as input" moves the result into the input box so you can run another tool on it, and your pasted text or your file on disk is never modified.

Try it on the Text Studio, and the guides page covers the rest of the studios. The standard plan is free right now - $0/month, 100,000 credits every 30 days, no card required.