How to Remove Duplicate Lines from a Text File Using Python (Without Changing Their Order)

Have you ever worked with a text file containing hundreds or thousands of lines, only to discover that many entries are repeated?

This happens with exported lists, log files, datasets, email lists, copied text, and programming data.

You could remove duplicates manually, but that becomes frustrating when the file contains thousands of lines.

In this tutorial, we'll build a small Python script that removes duplicate lines while preserving the original order.

What we'll build

Our script will:

• Read text from a file.
• Find repeated lines.
• Keep the first occurrence of each line.
• Preserve the original order.
• Save the cleaned result to a new file.

No external Python packages are required.

1. Example input

Create a file called "input.txt":

apple
banana
orange
apple
grape
banana
mango

We want to remove the repeated entries without alphabetically sorting the list.

Expected output:

apple
banana
orange
grape
mango

Notice that the original order of the unique entries remains unchanged.

1. The Python script

Create a file called "remove_duplicates.py" and add this code:

`
from pathlib import Path

def remove_duplicate_lines(input_file, output_file):
source = Path(input_file)
destination = Path(output_file)

if not source.is_file():
    raise FileNotFoundError(
        f"Input file not found: {source}"
    )

seen = set()
unique_lines = []

with source.open("r", encoding="utf-8") as file:
    for line in file:
        key = line.rstrip("\r\n")

        if key not in seen:
            seen.add(key)
            unique_lines.append(line)

with destination.open(
    "w", encoding="utf-8", newline=""
) as file:
    file.writelines(unique_lines)

print(f"Original lines: {sum(1 for _ in source.open('r', encoding='utf-8'))}")
print(f"Unique lines: {len(unique_lines)}")
print(f"Saved cleaned file to: {destination}")

if name == "main":
remove_duplicate_lines(
"input.txt",
"output.txt"
)
`
The script uses a Python "set" to track lines that have already appeared. A list stores the unique lines in their original order.

Important: This version treats "Apple" and "apple" as different entries. It also preserves blank lines, except when identical blank lines appear more than once.

1. Run the script

Make sure "input.txt" and "remove_duplicates.py" are in the same folder.

Run:

python remove_duplicates.py

The script creates "output.txt" containing the cleaned text.

If your system uses the "python3" command, run:

python3 remove_duplicates.py

1. How does it work?

"Path()"

Python's "pathlib" module makes it easier to work with file paths.

"seen = set()"

The set stores lines that have already been encountered. Checking whether an item exists in a set is typically very efficient.

"unique_lines = []"

The list remembers the order in which unique lines first appeared.

"rstrip("\r\n")"

This removes line-ending characters for comparison without removing ordinary spaces from the text.

"writelines()"

The cleaned lines are written to a separate output file, leaving the original input file untouched.

1. What about large files?

For ordinary text files, this approach is simple and efficient. Its memory usage grows with the number of unique lines because the script stores those lines in memory.

For files too large to fit comfortably in memory, a different approach—such as an external sorting workflow or a disk-backed database—may be more suitable.

1. Can you use this for real-world data?

Yes. Similar logic can help clean:

• Repeated product IDs
• Duplicate usernames in exported lists
• Repeated lines in text logs
• Duplicate entries in simple datasets
• Copied lists containing repeated items

Always keep a backup before cleaning important data, and check whether differences in capitalization or whitespace should count as duplicates for your particular task.

Explore more free online tools

If you regularly work with text, conversions, calculations, or developer utilities, you can also explore "Cometflow Space" (https://cometflow-space-new.vercel.app/), a growing collection of online tools and resources.

Final thoughts

Removing duplicate lines is a small task, but automating it saves time when working with repetitive data.

The useful part isn't just removing duplicates—it's doing so predictably, preserving the original order, and leaving the source file unchanged.

What other small Python scripts do you use to automate everyday tasks?

python #beginners #programming #productivity