How to Remove Duplicates From a Python List

Removing duplicates from a list in Python can be accomplished in various ways, each suited to different situations based on the order preservation requirement, the type of elements in the list, and performance considerations. Let's explore five examples, detailing each approach's mechanism and showcasing its output.

1. Using a Set to Remove Duplicates

Code:

original_list = [1, 2, 2, 3, 3, 3]
unique_list = list(set(original_list))
print(unique_list)

Explanation:

A set is created from original_list. Sets automatically remove duplicate elements since a set in Python cannot contain duplicates.
The set is then converted back into a list.
This method does not preserve the original order of elements.

Output:

[1, 2, 3]

2. Using a Dictionary (Preserving Order)

Code:

original_list = ['a', 'b', 'a', 'c', 'b']
unique_list = list(dict.fromkeys(original_list))
print(unique_list)

Explanation:

A dictionary is created from original_list using fromkeys(). This method removes duplicates because dictionary keys are unique.
The order of elements is preserved because dictionaries in Python 3.7+ are ordered.
The dictionary keys are then converted back into a list.

Output:

['a', 'b', 'c']

3. Using List Comprehension (Preserving Order)

Code:

original_list = [1, 2, 2, 3, 4, 4, 5]
unique_list = []
[unique_list.append(x) for x in original_list if x not in unique_list]
print(unique_list)

Explanation:

An empty list unique_list is initialized.
A list comprehension iterates over original_list. For each element, it checks if the element is not already in unique_list. If it's not, the element is appended to unique_list.
This method preserves the original order.

Output:

[1, 2, 3, 4, 5]

4. Using the `collections.OrderedDict` (For Older Python Versions)

Code:

from collections import OrderedDict
original_list = ['apple', 'banana', 'apple', 'pear', 'banana']
unique_list = list(OrderedDict.fromkeys(original_list))
print(unique_list)

Explanation:

OrderedDict.fromkeys() is used to create an ordered dictionary where all duplicate original_list items are removed.
The order of elements is preserved.
This method is useful for versions of Python older than 3.7, where regular dictionaries do not guarantee order preservation.

Output:

['apple', 'banana', 'pear']

5. Using a Loop (Preserving Order)

Code:

original_list = [10, 20, 20, 30, 40, 40, 50]
unique_list = []
for item in original_list:
    if item not in unique_list:
        unique_list.append(item)
print(unique_list)

Explanation:

unique_list is initialized as an empty list.
The code iterates through original_list. For each element, it checks if the element is not already in unique_list. If it's not, the element is appended.
This method preserves the original order and is straightforward but may not be the most efficient for large lists due to the if item not in unique_list check.

Output: