## DEV Community

jess unrein

Posted on • Updated on

# 3 Tricky Python Nuances

This blog post is adapted from a talk I gave last week at ChiPy, the Chicago Python User Group. For more great content like this, I highly recommend your local Python meetup, and PyVideo, the community project aimed at indexing every recorded Python meetup or conference talk.

# Nuance Number One

## Lists Don't Contain Objects

I know this sounds like a ridiculous statement. Everything in Python is an object. If declare a list, I'm filling it full of objects. Collecting objects is what a list does!

Kind of. Let me back up a bit.

Over Advent of Code a couple friends and I were sharing code snippets and debugging things together in our Friendship Discord. One of the problems involved creating a giant matrix that represented cloth, and you had to figure out which squares of cloth were being used by manipulating individual elements in that matrix. I have a friend who is learning Python right now, and setting the problem up turned out to be more difficult than they thought.

Here's how they set up their matrix initially. Simple enough, right?

``````
l = [[0] * 5] * 5

print(l)
[[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]]
``````

So far so good. But when they went to change an element in the first nested list...

``````l[0][3] = 1

print(l)
[[0, 0, 0, 1, 0],
[0, 0, 0, 1, 0],
[0, 0, 0, 1, 0],
[0, 0, 0, 1, 0],
[0, 0, 0, 1, 0]]
``````

๐ง

They posted in our discord group, asking what the heck was going on. My answer, as usual, was pretty much to shrug and ask why they weren't using a list comprehension. Another of our friends said something to the effect of "seems like it could be pointers?"

My second friend was right. Python doesn't store actual objects inside a list. It stores `references` to objects. And it's hella weird.

A `reference` points to a specific location in memory. That specific location contains the object, and if you change the object stored at that location in memory, you change every instance where it appears.

So each of the five lists inside the original matrix declared by `l = [[0] * 5] * 5` are actually just five separate `references` to the same list. When you change one, you change them all.

### So wait, how do I make them different objects?

Great question! In order to accomplish the goal of having a matrix where each element can be manipulated independently without inadvertently affecting other elements, we need to make sure we're creating a new object at a new location in memory for each nested list. In this case, we can use the copy library to create a new container object that we can change independently.

``````from copy import copy
a = [0, 0, 0, 0, 0]

b = []

# intentionally verbose example to illustrate a point :)
for _ in range(5):
b.append(copy(a))
``````

But of course, we can make this much cleaner. I was right the first time. Mooooooost of your Python problems can be solved with a well placed list comprehension (satisfaction not guaranteed).

``````l = [[0 for i in range(5)] for j in range(5)]

l[0][3] = 1

print(l)

[[0, 0, 0, 1, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]]
``````

# Nuance Number Two

## Default arguments are evaluated when the function is defined, not each time a function is called

What does this mean? It's easier to explain through an example. Let's pretend that, for some reason, we want to write a function that takes in an element and appends it to an existing list without using the `append` function.

``````def list_append(element, input_list=[]):
input_list.extend([element])
return input_list
``````

Now, this is a completely contrived example that we would never implement in practice, but it's useful for demonstration. Let's try calling that function a few times.

``````print(list_append(3))
[3]
print(list_append([5, 7]))
[3, 5, 7]
print(list_append("asdf"))
[3, 5, 7, "asdf"]
``````

When we call the function multiple times, it seems that it's always adding to and returning the same object, rather than starting with a fresh empty list each time the function is called.

This is because the default argument for a function is created and evaluated when the function is first declared. So it would get evaluated and you'd start with a fresh list every time you restarted your app, for example. But each time your function is called while the app is running it's going to be working off of the same mutable object, and will just keep growing and growing until the app is restarted again.

Here's how we fix this:

``````def list_append(element, input_list=None):
if not input_list:
input_list = []
return input_list.extend([element])

print(list_append(3))
[3]
print(list_append([5, 7]))
[5, 7]
print(list_append("asdf", ["bcde"]))
["bcde", "asdf"]
``````

Elements declared within a function's scope are evaluated every time the function runs, so in the above example you'll be starting with a new empty list every single time, instead of passing the same default argument from function call to function call.

This isn't just a problem with lists, by the way. You should always be careful about passing in mutable objects as default arguments to a function; all mutable objects will have this same problem.

# Nuance Number Three

## `==` vs `is`

`is` and `==` are both operators in Python that evaluate two objects against one another and return a boolean value based on the results. This is a little tricky, but `==` checks for equality while `is` checks for identity.

Remember back to nuance number one when we talked about `references`, or specific locations in memory that contain an object? They're important again here!

Equality means that the values of two objects match. Identity means that the two things you're comparing are the exact same object and exist at the same location in memory. You can check the location of an object in memory by using the builtin Python function `id()`. So when you compare two objects using the `is` operator, you're checking that the two objects have the same `id`, not the same `value`.

A quirk of this, that you'll often see in ideomatic Python, is that there is only one `True` boolean object and only one `False` boolean object in memory. Each appearance of `True` has the same id as every other appearance of `True`. However, there are some values that evaluate as being equal to `True` without sharing its identity. That's why Pythonistas often use the `is` operator when comparing things to `True` or `False` rather than the `==` operator.

This can be a bit confusing and jargon filled when explained in words. Here are some examples.

### Equality vs identity for boolean values:

``````print(1 == True)
True
print(1 is True)
False
a = True
print(a is True)
True

print(0 == False)
True
print(0 is False)
False
b = False
print(b is False)
True
``````

### Equality vs identity - equivalent variables

``````# a and b variables point to the same id
a = [1, 2, 3, 4, 5]
b = a
print(a == b)
True
print(a is b)
True
print(id(a), id(b))
4437740104, 4437740104

# a and b are different container objects, but the values of their contents are identical
a = [1, 2, 3, 4, 5]
b = [1, 2, 3, 4, 5]
print(a == b)
True
print(a is b)
False
print(id(a), id(b))
4437740104, 4442640968
``````

# Conclusion

I'm not sure I have a unified conclusion here. These are just three things that might trip you up in Python that are hard to google for, since they lead to unexpected behaviors but don't give you a nice error message.

I suppose if I were to give you a takeaway it's that
1.) `references` are kind of wonky. You usually don't have to think about them, but they're worth understanding
2.) Always use list comprehensions

As always, let me know if you have any questions in the comments, or if you think I missed anything important regarding these three topics. Let me know if there's a Python quirk you don't get and would like an explainer on. Thanks!

Josรฉ Jorge

Good info!

I got caught by number 2 in my previous project, I was trying something like this:

``````def some_function(some_date = datetime.utcnow()):
...
``````

I'd get the same date everytime `some_function` was called.

Changed to something like this:

``````def some_function(some_date = None):
if some_date is None:
some_date = datetime.utcnow()

...
``````

Jason C. McDonald

Oof, that's a keeper!

Nicolas Viard

1) you should use a linter like pylint to avoid mistakes like this
2) is can be confusing for small integers because they share references
1 is 1 = True
158896587456 is 158896587456 = False

jess unrein • Edited

1.) I mean. I almost always find โyou should use a linter and then you can avoid learningโ to be an unhelpful piece of advice. Also, a linterโs not going to tell you why a nested array is behaving unexpectedly.
2) And true. Python can definitely get a little weird around the edges when you get to arbitrarily large numbers. I hope youโre not hardcoding and checking for identity for integers like this in your code.

You clearly understand references very well! These short examples are, of course, aimed at introducing a concept and encouraging people to learn more. Thanks for providing some trivia!

Modaf

I tried 158896587456 is 158896587456 on Python 3.6.1 and it answers True :/
(Even with huge numbers it does answer True)

Nicolas Viard

Sorry, this is true when (at least) one of them is stored into a variable:

``````>>> a = 1
>>> b = 1
>>> print(a is b)
True
>>> a = 123456789
>>> b = 123456789
>>> print(a is b)
False
``````

Modaf

Indeed ! But it's weird that 123456789 is 123456789 = True, isn't it ? As if Python answers True for a is b if "a is the same as b" before testing if they are the same. Is there a reason for that ?

(123456789+0) is 123456789 = False
but 123456789 is 123456789 = True

Anyway thanks for the example and the answer !

Optimizer gets better and better. In 3.7, even 123456788+1 is 123456789 gives True. It's simply that Python realizes it can evaluate constant pure expressions before compiling to bytecode. And it folds constants, of course.

`>>> compile('123456788+1 is 123456789', '', 'eval').co_consts`

You'll notice there's only one 123456789 there, not two. And there are no 123456788 nor 1.

Good points about lists and default argument. I disagree with your take on `is` v `==`. It is an anti-pattern to test booleans with `is` or `==`. They are already booleans, so there is no reason to write `if x is True:`, just write `if x:`.

Also, Python is happy to use any value in boolean contexts, which makes it even less common to handle booleans explicitly as you suggest.

The only really common case of using `is` in Python code is to test for `None`, as in `if x is None:`. This is clear, explicit, and performant, because `None` is singleton, and there is no overloading for the `is` operator, so it is faster than `==` which can be overloaded via the `__eq__` special method. Other than that, uses cases that actually require or recommend the use of the `is` operator are extremely rare.

jess unrein

I have worked in a couple teams where my engineering managers had a real stick in their craw about always using `is` comparators with booleans to the point of rejecting PRs that didn't follow this pattern. But you may be right that the more Pythonic way would be to write `if x:` when checking for Truth. It's a pattern that I've had drilled into my head that might, in fact, not be as idiomatically Pythonic as I had been led to believe.

However, to your point, if you want to check that something is explicitly `False` (as opposed to something being `False` or `None`) it makes sense to me to use an explicit comparator rather than relying on `if not x:` as `if not` reads `None` as a falsey value. Granted, the use case here is pretty narrow, so it's not super useful.

Thank you for the clarification!

kaelscion

My favorite quote I've ever heard about Python is this: "Just like Stephen Wright, Python's greatest strength is how well it does one-liners" and I shamelessly use it all the time in posts and comments ๐. List, dict, set, and generator comprehensions are great strengths even when compared with lambdas or zip() functions. Great article as always Jess!๐๐

Aidas Bendoraitis

Great article. Makes sense.

However, it looks like you posted the `list_append()` examples without testing them beforehand, for example, these are equivalents:

``````print(list_append(5, 7))
print(list_append(element=5, input_list=7))
``````

and you cannot call `.extend()` on an integer.

jess unrein • Edited

Oops, this is my bad retyping something from a screenshot of a slide, so I couldn't copy paste. Thanks for the catch! It should read `print(list_append([5, 7]))`, as the example below it does, and I've amended the text to reflect this.

Peter

Wow, thanks for the article and your concise explanations! :) I didn't know the first nuance before. And it's by sheer luck that it didn't hurt my work already, since I used the * notation for lists before - luckily without changing any of the referenced objects afterwards.

And I nearly forgot about the second nuance (shame on me).

Tom VanAntwerp

I'm certain #2 (and maybe in concert with #1) cost me hours of my life before I learned what was happening. Great list of gotchas!

Modaf

Have learned something, ty

Michael

Thanks for the great article!

Jason C. McDonald

Great insight. I cross-linked to this at the end of Dead Simple Python: Data Types and Immutability.

Jason C. McDonald