How to remove duplicates from Python List

Removing duplicates from a list is a common task in Python programming. Whether you’re working with data analysis, web development, or any other field, you’ll often encounter lists containing duplicate values that need to be eliminated.

This article provides a comprehensive guide to several effective methods for removing duplicates from a list in Python. We’ll explore different approaches, from using sets to more advanced techniques, along with clear code examples and explanations to help you choose the best method for your specific needs. We will cover list comprehension, the dict.fromkeys() method, and more. So, you’ll master Python list de-duplication with ease. Let’s get started!

Example Input/Output

Let’s start with an example list that contains duplicates. We’ll demonstrate how to remove these duplicates using various methods.

 Input List: [1, 2, 2, 3, 4, 4, 5]
 Output List (without duplicates): [1, 2, 3, 4, 5]
 

Method 1: Using Sets to Remove Duplicates

One of the simplest and most efficient ways to remove duplicates from a list is by converting it to a set. Sets, by definition, only contain unique elements. Once the duplicates are removed, you can convert the set back into a list if needed.

 def remove_duplicates_with_sets(input_list):
     """Removes duplicates from a list using sets."""
     return list(set(input_list))

 my_list = [1, 2, 2, 3, 4, 4, 5]
 unique_list = remove_duplicates_with_sets(my_list)
 print(f"Original List: {my_list}")
 print(f"List with Duplicates Removed (using sets): {unique_list}")
 

 Original List: [1, 2, 2, 3, 4, 4, 5]
 List with Duplicates Removed (using sets): [1, 2, 3, 4, 5]
 

Explanation: The code defines a function remove_duplicates_with_sets() that takes a list as input. Inside the function, the input list is converted into a set using set(input_list). This operation automatically removes all duplicate elements, as sets only store unique values. The resulting set is then converted back into a list using list(). Finally, the function returns the new list containing only unique elements. This method is very fast and memory-efficient, especially for large lists, because sets are implemented using hash tables.

Method 2: Using List Comprehension (Preserving Order)

If preserving the original order of elements in the list is important, using list comprehension with a conditional check can be a good approach. This method iterates through the list and only adds elements to a new list if they haven’t been encountered before.

 def remove_duplicates_with_list_comprehension(input_list):
     """Removes duplicates from a list while preserving order using list comprehension."""
     seen = set()
     unique_list = [x for x in input_list if x not in seen and not seen.add(x)]
     return unique_list

 my_list = [1, 2, 2, 3, 4, 4, 5, 1, 6, 7, 7]
 unique_list = remove_duplicates_with_list_comprehension(my_list)
 print(f"Original List: {my_list}")
 print(f"List with Duplicates Removed (using list comprehension, preserving order): {unique_list}")
 

 Original List: [1, 2, 2, 3, 4, 4, 5, 1, 6, 7, 7]
 List with Duplicates Removed (using list comprehension, preserving order): [1, 2, 3, 4, 5, 6, 7]
 

Explanation: The function remove_duplicates_with_list_comprehension() takes a list as input. It initializes an empty set called seen to keep track of the elements already encountered. The list comprehension [x for x in input_list if x not in seen and not seen.add(x)] iterates through the input list. For each element x, it checks if x is not in the seen set. If it’s not, the element is added to the seen set using seen.add(x) (which also returns None, ensuring the element is included in the result), and the element x is included in the new list. This ensures that elements are only added to the new list if they haven’t been seen before, thus preserving the original order while removing duplicates.

Method 3: Using dict.fromkeys() (Preserving Order – Python 3.7+)

Starting from Python 3.7, the insertion order of dictionaries is guaranteed. This allows us to use dict.fromkeys() to remove duplicates while preserving the order. This method is generally faster than list comprehension, especially for larger lists.

 def remove_duplicates_with_dict_fromkeys(input_list):
     """Removes duplicates from a list while preserving order using dict.fromkeys()."""
     return list(dict.fromkeys(input_list))

 my_list = [1, 2, 2, 3, 4, 4, 5, 1, 6, 7, 7]
 unique_list = remove_duplicates_with_dict_fromkeys(my_list)
 print(f"Original List: {my_list}")
 print(f"List with Duplicates Removed (using dict.fromkeys(), preserving order): {unique_list}")
 

 Original List: [1, 2, 2, 3, 4, 4, 5, 1, 6, 7, 7]
 List with Duplicates Removed (using dict.fromkeys(), preserving order): [1, 2, 3, 4, 5, 6, 7]
 

Explanation: The function remove_duplicates_with_dict_fromkeys() uses the dict.fromkeys() method to create a dictionary with elements from the input list as keys. Since dictionaries cannot have duplicate keys, this automatically removes duplicates. Converting the keys of this dictionary back into a list preserves the original order of the unique elements. This method is generally faster and more concise than list comprehension, especially for larger lists, due to the efficient implementation of dictionaries in Python.

Method 4: Using OrderedDict (Preserving Order – Python 3.6 and Earlier)

If you’re using Python 3.6 or earlier, where dictionary insertion order is not guaranteed, you can use OrderedDict from the collections module to achieve the same result as dict.fromkeys() in later versions.

 from collections import OrderedDict

 def remove_duplicates_with_ordered_dict(input_list):
     """Removes duplicates from a list while preserving order using OrderedDict (for Python 3.6 and earlier)."""
     return list(OrderedDict.fromkeys(input_list))

 my_list = [1, 2, 2, 3, 4, 4, 5, 1, 6, 7, 7]
 unique_list = remove_duplicates_with_ordered_dict(my_list)
 print(f"Original List: {my_list}")
 print(f"List with Duplicates Removed (using OrderedDict, preserving order): {unique_list}")
 

 Original List: [1, 2, 2, 3, 4, 4, 5, 1, 6, 7, 7]
 List with Duplicates Removed (using OrderedDict, preserving order): [1, 2, 3, 4, 5, 6, 7]
 

Explanation: This method is very similar to using dict.fromkeys(). The key difference is that it uses OrderedDict from the collections module, which guarantees the order of insertion is preserved. The OrderedDict.fromkeys() method creates an ordered dictionary where the elements of the input list become the keys, effectively removing duplicates while retaining the order. Finally, converting the keys of the OrderedDict back into a list gives the desired result.

Method 5: Using a Loop and a Set

This is a more verbose but clear approach that uses a loop to iterate through the original list and a set to keep track of seen elements.

 def remove_duplicates_with_loop(input_list):
     """Removes duplicates from a list using a loop and a set, preserving order."""
     seen = set()
     unique_list = []
     for item in input_list:
         if item not in seen:
             seen.add(item)
             unique_list.append(item)
     return unique_list

 my_list = [1, 2, 2, 3, 4, 4, 5, 1, 6, 7, 7]
 unique_list = remove_duplicates_with_loop(my_list)
 print(f"Original List: {my_list}")
 print(f"List with Duplicates Removed (using a loop, preserving order): {unique_list}")
 

 Original List: [1, 2, 2, 3, 4, 4, 5, 1, 6, 7, 7]
 List with Duplicates Removed (using a loop, preserving order): [1, 2, 3, 4, 5, 6, 7]
 

Explanation: The function initializes an empty set called seen to keep track of encountered elements and an empty list called unique_list to store the unique elements. The code then iterates through the input list using a for loop. For each item in the input list, it checks if the item is already present in the seen set. If the item is not in the seen set, it means it’s the first time we’re encountering this element. In that case, the item is added to the seen set using seen.add(item) and appended to the unique_list using unique_list.append(item). By the end of the loop, the unique_list will contain only the unique elements from the input list, preserving their original order.