A list picks up duplicates the moment more than one person touches it. You export email addresses from two places and merge them, and half the contacts appear twice. You combine keyword lists from three campaigns into one. You copy names out of a chat thread into a roster and the same people show up three times. Duplicates are not just untidy: an email list with repeats sends the same person the same message twice, a keyword list with repeats wastes every check you run against it, and a roster with repeats makes a headcount wrong. Cleaning the list by hand works for twenty lines and breaks down around two hundred, because the eye skips repeats it has already seen. The Remove Duplicate Lines tool does the job in one paste, and this guide covers the method, the three settings, and the invisible characters that stop two identical-looking lines from matching.
The method is short. Paste the list into the Remove Duplicate Lines tool’s Lines box and the cleaned version appears in the Unique lines box as you type, with a counter underneath reading something like 5 lines in → 3 lines out, so you can see at a glance how many duplicates were removed. Set the three checkboxes first if they matter to your list, then copy the result out with the Copy button and paste it back into your document, spreadsheet or mailing tool. The rest of this guide is about choosing those three settings correctly, because the defaults suit most lists but not all of them.
The three settings, and when to change them
Three checkboxes sit between the two boxes, and each one changes the result in a specific way. Case sensitive is ticked by default, so lines are compared exactly as typed. Keep blank lines is unticked, so empty lines are stripped out. Keep last occurrence is unticked, so the first copy of each repeated line is the one that survives. For a plain list of names or keywords, leave all three alone. Change them only when you know why: untick Case sensitive when capitalisation should not create phantom duplicates, tick Keep blank lines when blank lines separate sections you want to keep apart, and tick Keep last occurrence when the newest copy of a line is the one with the corrected content.
- Case sensitive (ticked by default): lines are compared exactly as typed. Apple and apple are different lines and both survive.
- Keep blank lines (unticked by default): blank lines are removed. Tick it and a blank line survives as a separator, but repeated blanks still collapse to one.
- Keep last occurrence (unticked by default): the first copy of a repeated line survives. Tick it and the newest copy wins instead.
Worked example: five lines in, three lines out
The tool opens with a sample list already loaded, and it is worth watching what happens to it before you paste your own list. Five lines go in: apple, banana, apple, cherry, banana. Three lines come out: apple, banana, cherry. The second apple and the second banana are gone, the first copies stay exactly where they were, and nothing else changes — the surviving lines keep their original order and their original wording. That order-preserving behaviour is deliberate: order carries meaning in logs, rosters, instructions and ranked lists, so the tool never reorders anything. If you also need alphabetical order, dedupe first and sort afterwards in a spreadsheet, where the sort has less work to do and no repeats left to hide.
Try it with the sample: apple, banana, apple, cherry, banana goes in as 5 lines and comes out as apple, banana, cherry — 3 lines. Paste your own list over the sample and the counter updates as you type.
Exact or case-insensitive: when Ali and ali are the same entry
Case is the setting that catches most people out, because it decides what counts as the same line. With Case sensitive ticked — the default — Apple, apple and APPLE are three different lines and all four lines of Apple, apple, APPLE, Banana survive. Untick it and the three spellings share one identity, so only the first spelling is kept and the same four lines come back as just Apple and Banana. The surviving copy keeps the spelling of the first occurrence, which is the detail to check when the copies differ only in capitals: the tool never invents a spelling, it keeps the one it saw first.
Untick the option for email lists and keyword lists, where capitalisation should not create phantom duplicates — the same address typed as Ali@example.com in one export and ali@example.com in another is one person, and the same keyword in two cases is one keyword. Leave it ticked for anything where case carries meaning: codes, passwords and case-sensitive identifiers, where ABC123 and abc123 are genuinely different entries. If a list mixes the two — mostly names, but a few codes — normalise the case first with the Case Converter, which switches a whole list to lower, upper or title case in one pass, and then dedupe with the case setting left on.
Blank lines and the spaces you cannot see
Blank lines are removed by default, including lines that contain only spaces, which makes the tool handy for tightening up text pasted from PDFs and web pages. If you tick Keep blank lines, a blank line is allowed to survive as a separator between sections, but repeated blank lines are still treated as duplicates of each other, so a gap of three empty lines comes back as a single blank line. A list of alpha, then three blank lines, then beta returns as alpha and beta with the blanks stripped, or as alpha, one blank line, beta with the option ticked. The counter underneath counts every line you paste, blank ones included, so a list with three blank lines reads as more lines in than the cleaned result suggests.
The harder problem is the space you cannot see. The comparison is exact and nothing is trimmed, so a line with a trailing space and the same line without one look identical on screen and are treated as different lines: apple followed by a space, then apple on the next line, both survive. The same applies to a tab character or a non-breaking space copied from a web page, which is invisible in most editors. If a line refuses to dedupe when you can see it twice, look for trailing spaces first — retype the line by hand, or trim the spaces in the source and paste again. Paste the before and after versions into the Text Diff Checker if you want to see exactly which lines were dropped and which survived.
Keep first or keep last: which copy survives
The default keeps the first occurrence of each line in the position where it first appeared, which is the right choice for most lists: the earliest entry is usually the original, and its position is where the reader expects it. Tick Keep last occurrence and the newest copy wins instead, which suits data where the same entry was written again later with updated content, such as a re-registration or an edited settings line. The set of unique lines is the same either way; only which copy survives, and where it sits, changes. With case-insensitivity on, Apple, banana, apple becomes Apple, banana with the first occurrence kept, and banana, apple with the last occurrence kept — the later spelling survives and the line sits where its last copy was.
Doing it in a spreadsheet instead
Excel and Google Sheets both have a Remove Duplicates command — in Excel it sits on the Data tab, in Sheets under Data and then Data cleanup — and a spreadsheet is the better tool when the list lives inside a workbook with other columns, because deduping a column in place keeps the rows aligned with the data beside them. For a plain list of lines, though, the spreadsheet route is slower: open a workbook, paste the list, find the command, answer the prompt about headers, then copy the result back out. The online tool is one paste for a plain list, with the result updating as you type and no file to open or save. Use the spreadsheet when the list has neighbours; use the tool when the list is the whole job. Either way, dedupe before you sort, import or send — duplicates caught late cost more than duplicates caught at the paste.
Why the list never leaves your browser
An email list is customer data and a keyword list is business data, and both deserve more care than a random upload box on a stranger’s server. This tool splits the text on line breaks and compares each line against the ones already seen entirely in your browser, so nothing is uploaded, stored or logged, and a private list can be cleaned safely. That is the same on-device rule the whole site runs on — every tool here works the same way, which is why the privacy guide can make the claim for all of them at once. If a dedupe tool asks you to create an account before it will clean a list, that is a sign the list is the product, and you should paste it somewhere else. Browse the rest of the text tools for its companions: the case converter, the text diff checker and the word counter.
Try it now: paste the list into the Remove Duplicate Lines tool, check the counter, and copy the cleaned result back. Five minutes of checking beats an afternoon of untangling a mailing list that went out twice.
Frequently asked questions
How do I remove duplicate lines from a list online?
Paste the list into the Lines box of the Remove Duplicate Lines tool and the cleaned version appears in the Unique lines box as you type, with a counter showing lines in and lines out. Set the three options first if they matter: untick Case sensitive to merge lines that differ only in capitals, tick Keep last occurrence if the newest copy should win, and tick Keep blank lines if you want one blank separator preserved. Then copy the result out with the Copy button.
Does it treat Apple and apple as the same line?
Only if you untick Case sensitive. With case sensitivity on, the default, Apple, apple and APPLE are three distinct lines and all survive. With it off, they share one identity and only the first spelling is kept, so Apple, apple, APPLE and Banana comes back as just Apple and Banana. Untick the option for email lists and keyword lists, where capitalisation should not create phantom duplicates.
What happens to blank lines?
By default blank lines are removed, including lines that contain only spaces, which makes the tool handy for tightening up text pasted from PDFs and web pages. If you tick Keep blank lines, a blank line is allowed to survive as a separator between sections, but repeated blank lines are still treated as duplicates of each other, so you get one blank line rather than a run of them.
Should I keep the first or the last occurrence?
Keep the first occurrence for most lists: it preserves the earliest copy of a repeated line in the position where it first appeared. Keep the last occurrence when the same entry was written again later with updated content, such as a re-registration or an edited settings line. The set of unique lines is the same either way; only which copy survives, and where it sits, changes.
Will it sort my list alphabetically at the same time?
No. This tool only removes duplicates, and deliberately keeps the surviving lines in their original order, because order carries meaning in logs, rosters, instructions and ranked lists. If you also need alphabetical order, copy the cleaned list into a spreadsheet or text editor and sort it there; deduping first means the sort has less work to do and no repeats to hide.
Top comments (0)