Easy Way to Find Duplicates in a List Using Sets in Python

Updated November 1, 2023

Finding duplicates in a list can be a bit of a pain. However, it’s a piece of cake once you use the built-in set data structure in Python because sets only contain unique elements. Here is a simple way to detect and remove duplicates from a list by converting the list into a set, then back into a list.

# Python program to demonstrate conversion 
# of list to Set and vice versa

# List with duplicate values
list1 = [10,20,30,40,50]

# Converting list to set
set1 = set(list1)
print("The set after converting list is :", set1)

# Back conversion of set to list
list2 = (list(set1))
print("The list after back conversion of set :", list2)

In this example, the duplicate values are not present in the converted set. This is because sets only contain unique elements by definition.

Now, let’s consider a situation where you have a list with duplicated data and you want to find out which items appear more than once in the list. Here we will use collections.Counter():

# Python program to demonstrate Counter 
import collections

# List with duplicate values
list1 = [10,20,30,40,50,40]

# Converting list to Counter object
counter_obj = collections.Counter(list1)
print("The counter after converting list is :", counter_obj)

# To check the items that appear more than once in a list
for item, frequency in counter_obj.items():
    if frequency > 1:
        print('Duplicate value:', item)

In this example, you can see that we have two “40” elements in our original list and that’s why it shows up as a duplicate when we use collections.Counter().