One Change Made My Python Code 2,000 Faster (Lists vs Sets)

One tiny change made this Python code over 2,000× faster. Same logic, same result. The only difference is a list versus a set.

Here's the video version (2 minutes):

The slow code

A common task: 20,000 users, a list of 20,000 banned emails, find which users are banned.

users  = [...]   # 20,000 emails
banned = [...]   # 20,000 banned emails (a list)

flagged = [u for u in users if u in banned]

One line, looks perfectly fine. On my machine it took 1.31 seconds.

The fix: one function call

banned = set([...])   # a set instead of a list

flagged = [u for u in users if u in banned]

The comprehension doesn't change at all, and it returns the same 10,000 results. It now takes 0.61 milliseconds. That's over 2,000× faster, and building the set took less than a millisecond.

Why: how in works

• x in list checks the items one by one until it finds a match → O(n)
• x in set hashes x and jumps straight to the slot where it would be stored → O(1) on average
• Dictionary keys work the same way: key in my_dict is O(1)

With lists, our example is up to 20,000 × 20,000 = 400 million comparisons. With a set, it's 20,000 hash lookups.

Measured: a single lookup at different sizes

Worst case (the item is at the end of the list):

n =     1,000   list     4.3 µs   set  0.021 µs
n =   100,000   list   455.0 µs   set  0.027 µs
n = 1,000,000   list  4560.7 µs   set  0.027 µs

The list gets ~100× slower when the data gets 100× bigger. The set barely moves. At a million items, a set lookup was over 170,000× faster.

When to use a set (and when not to)

✅ Use a set when:

• You check membership (in) many times against the same collection
• You want to remove duplicates: set(items)
• You need intersections or differences: a & b, a - b

❌ Keep a list when:

• You need order, indexes, or duplicates
• You only look something up once: converting to a set costs more than it saves

Try it on your code

Search your codebase for in followed by a big list inside a loop. You might find a free speedup.

---

I post one tested coding tip every day on YouTube at Refactorly. The 30-second version of this one is here.