Mutable Default Arguments in Python

This article was originally shared on my blog.

Objects of built-in types like (int, float, bool, str, tuple, Unicode) are immutable. Objects of built-in types like (list, set, dict) are mutable.
A mutable object can change its state or contents and immutable objects cannot.

This is a very simple definition of a Mutable and Immutable object in Python. Now coming to the interesting part which is Mutable Default Object in function definitions.

I have written a very simple function as below -

def foobar(element, data=[]):
    data.append(element)
    return data

In the above function, I set data to a list(Mutable object) as a default argument. Now let's execute this function -

>>> print(foobar(12))
[12]

The output is as expected, Now execute multiple times -

>>> print(foobar(22))
[12, 22]
>>> print(foobar(32))
[12, 22, 32]

What is going on here? As you can see, the first time, the function returns exactly what we expected. On the second and all subsequent passes the same list is used in each call.

Why is this happening

Python’s default arguments are evaluated once when the function is defined, not each time the function is called. When Python encounters it, the first thing it will do is compile it in order to create a code object for this function. While this compilation step is done, Python evaluates and then stores the default arguments in the function object itself.

So, let's do some introspection, to clear the confusions, I have taken a few lines of code as below-

• Function Before Execution -

def foo(l=[]):
    l.append(10)

After function executes this definition, You can check the default attribute for this function. by using __defaults__.

About __defaults__
A tuple containing default argument values for those arguments that have defaults, or None if no arguments have a default value.

>>> foo.__defaults__
([],)

The result an empty list as the only entry in __defaults__

• Function After Execution-

Let's now execute this function and check the defaults.

>>> foo()
>>> foo.__defaults__
([10],)

Astonished? The value inside the object changes, Consecutive calls to the function will now simply append to that embedded list object:

Let's execute the function multiple times:

>>> foo()
>>> foo()
>>> foo()

>>> foo.__defaults__
([10, 10, 10, 10],)

The reason why this is happening because default argument values are stored in the function object, not in the code object (because they represent values calculated at run-time).

The Solution

Now, the question is how to code foobar in order to get the behavior that we expected.

Fortunately, the solution is straightforward. The mutable objects used as defaults are replaced by None, and then the arguments are tested for None.

def foobar(element, data=None):
    if data is None:
        data = []
    data.append(element)
    return data

>>> foobar(12)
[12]

So, by replacing mutable default arguments with None solved our problem.

Good Use of Mutable Default Argument

However, Mutable Default Argument have some good use-case also as following-

1.Binding local variable to the current value of the outer variable in a callback
2.Cache/Memoization

I hope you like the explanation of Mutable Default Arguments in Python. Still, if any doubt or improvement regarding it, ask in the comment section.

References:
https://docs.python.org/3/reference/datamodel.html#the-standard-type-hierarchy

Comments

[rhymes]

Great question! I had to refresh my knowledge a bit about the why

Nice, I see this as a great time to ask though, is there an Object.freeze() equivalent in Python that will automatically let us make mutable objects immutable?

Not really, there are immutable objects like numbers, strings, tuples and frozen sets but a generic object can't be automatically frozen.

You can play with some builtin methods like setattr (which is called when a new attribute is added to an object) and its specular getattribute to create a custom immutable object

And if not, why wouldn't something like that be added?

Probably because the language has immutable data types that are used daily: numbers, strings and tuples.

For example:

>>> a = "abc"
>>> a[0] = "x"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'str' object does not support item assignment

but in Ruby:

2.7.0 :001 > a = "abc"
2.7.0 :002 > a[0] = "x"
2.7.0 :003 > a
 => "xbc"

if you look at many Ruby code bases, you'll see .freeze used beside many strings that are defined as constant because by default they are mutable. In Python they aren't. I personally prefer the latter design choice.

In 2005 there was a PEP (the name of Python community proposals) to add a freeze() method, called freeze protocol. It was rejected and among the reasons there are the following:

• the concept of frozen objects in Python is related to them being able to be keys in a dictionary (all immutable objects can be), but why would a dev need an entire object to be a key in a dictionary?

• sending objects around both in their mutable version and their immutable one can lead to errors. Ruby has a .frozen? method to help you distinguish them, but still, that would mean that each receiver of that object would need to check if the object is frozen or not. It's easier to assume that all objects aside from those immutable builtins are mutable and live with that.

TLDR; language design choices. 15 years have passed since that proposal and Python doesn't seem to be in need of a freeze() method after all

[rhymes]

Citing Guido Van Rossum in the response about the freeze protocol mentioned in the other comment:

Even if you skip past the
question of why you would want to freeze a dictionary (do you really want to
use it as a key?), one find that dicts are not naturally freezable -- dicts
compare using both keys and values; hence, if you want to hash a dict, you
need to hash both the keys and values, which means that the values have to
be hashable, a new and suprising requirement -- also, the values cannot be
mutated or else an equality comparison will fail when search for a frozen
dict that has been used as a key.

and then

One person who experimented with an
implementation dealt with the problem by recursively freezing all the
components (perhaps one of the dict's values is another dict which then
needs to be frozen too).

and then

Executive summary: freezing dicts is a can of worms and not especially useful.

from mail.python.org/pipermail/python-d...

If we take a step back we can understand this ourselves. In Python dict keys need to be immutable but their values don't. So what "freezing a dict" might mean? Not being able to add new key values pairs? Maybe that's achievable by playing with a subclass of collections.UserDict but if any of the values of the dict are objects themselves, you can still change the object and thus you'll end up passing around a "frozen" hash that's not actually constant throughout its life.

Basically, a can of worms :D

Object.freeze({a: 3}) in JS will return a frozen object BUT if the I swap 3 for an object in another variable, I can actually change that value and the result is that your frozen object might not allow you to add new keys or assign to it, but its actually value is still not the same of when you frozen it:

const anotherObj = {d: 4};
undefined
const x2 = Object.freeze({a: anotherObj})
undefined
x2
{…}
​
a: {…}
​​
d: 4
​​
<prototype>: Object { … }
​
<prototype>: Object { … }

anotherObj
Object { d: 4 }

anotherObj.d = 5
5
x2
{…}
​
a: Object { d: 5 }
​
<prototype>: Object { … }

sorry for the mess but I copy and pasted quickly from the browser's console :D

Same happens with Ruby BTW:

[26] pry(main)> obj = {a: 3}
=> {:a=>3}
[27] pry(main)> frozen = {b: obj}.freeze
=> {:b=>{:a=>3}}
[28] pry(main)> frozen[:c] = 10
FrozenError: can't modify frozen Hash
from (pry):28:in `<main>'
[29] pry(main)> obj[:a] = 10
=> 10
[30] pry(main)> frozen
=> {:b=>{:a=>10}}

so yeah, freeze is a neat idea but not very useful in a language like Python (nor very useful in Ruby beyond making strings immutable and other few cases)

[rhymes]

Honestly? I don't think so.

It might be better to use a functional approach and limit sharing data. For example, using Ruby:

2.6.5 :001 > a = {key: "value"}
 => {:key=>"value"}
2.6.5 :002 > b = a.merge(foo: :bar)
 => {:key=>"value", :foo=>:bar}
2.6.5 :003 > a
 => {:key=>"value"}
2.6.5 :004 > b
 => {:key=>"value", :foo=>:bar}
2.6.5 :005 > def transform_data(data)
2.6.5 :006?>   data.merge(new: :data)
2.6.5 :007?>   end
 => :transform_data
2.6.5 :008 > c = transform_data(b)
 => {:key=>"value", :foo=>:bar, :new=>:data}
2.6.5 :009 > [a, b, c]
 => [{:key=>"value"}, {:key=>"value", :foo=>:bar}, {:key=>"value", :foo=>:bar, :new=>:data}]

as you can see I never change a or b. I just create new objects, I never change the content of the variable.

This is similar to how pure functional languages work or languages like Erlang where variables are immutable, so once they have a value, they'll keep that for their entire life.

[rhymes]

Python doesn't have constants