Odd Behavior when Appending a Python List to Another List

Written by Dan Sackett on October 22, 2014

Python can be funny sometimes in how it works behind the scenes. There are a lot of instances where you expect one result and get something completely wrong (to you).

Last night my brother in law Jesse and I were working on an odd Python result. The code we were trying to use was:

new_list = []
iter_list = range(16)
pos = [0]

for item in iter_list:
    pos[0] += 50
    new_list.append(pos)

print new_list
# [[800], [800], [800], [800], [800], [800], [800], [800], [800], [800], [800], [800], [800], [800], [800], [800]]

First off, let me define what we were expecting. We wanted to create a new list which would consist of single value lists incrementing by 50. So in essence, we wanted this:

[[50], [100], [150], [200], [250], [300], [350], [400], [450], [500], [550], [600], [650], [700], [750], [800]]

As you can see, something was going wrong. We were appending fine, but the issue was that our resulting list was a repeat of the last value. Naturally, we tried to do something a little different:

new_list = []
iter_list = range(16)
pos = 0

for item in iter_list:
    pos += 50
    new_list.append([pos])

print new_list

Of course, this works. The difference here is we define pos as an integer and increment this in each loop iteration. When we append, we append the new number inside a list and all works fine. So this led us to Stack Overflow. It turns out that in our initial example, we were actually appending a reference to our new list. Because of this, whenever we updated our pos variable, all references were also affected. This is why the iteration was correct but the values weren't.

So how do we solve this?

We can either take the integer approach as I mentioned above, or we have to make sure that we aren't appending a reference to our new list. There are two ways to avoid this.

The first way has us cast our value to a list as we append it:

new_list = []
iter_list = range(16)
pos = [0]

for item in iter_list:
    pos[0] += 50
    new_list.append(list(pos))

print new_list

By casting to a list, we create a new list. This new list is not bound to our variable so therefore it won't update as we increment the pos. The second way is by using the list copy syntax my_list[:].

new_list = []
iter_list = range(16)
pos = [0]

for item in iter_list:
    pos[0] += 50
    new_list.append(pos[:])

print new_list

This creates a new list through copying so this also will not update as we loop through. This subtlety in Python is something neither of us have come across before. It reminds me of this common Python pitfall though:

>>> def foo(bar=[]):
...     bar.append('baz')
...     return bar
... 
>>> foo()
['baz']
>>> foo()
['baz', 'baz']
>>> foo()
['baz', 'baz', 'baz']

So what's happening is we define a function and state an empty list as the default argument for our function. When we call our function, we append the word "baz" to the list as we expect. When we call it again, though, we use the same list as before. What you need to know here is that the default value for a function argument is only evaluated once, at the time that the function is defined. This means that the list bar is defined at the first run and then all other calls to it are by reference like the issue Jesse and I ran into.

To get around this, we can be a little smarter:

>>> def foo(bar=None):
...     if bar is None:
...         bar = []
...     bar.append('baz')
...     return bar
... 
>>> foo()
['baz']
>>> foo()
['baz']

In this updated example, our default argument is None. This way, we do a simple check if the item is None and if so then we define a new list.

Conclusion

In the end, be careful with references. When you are finding your values updating when you don't expect them too, check and see if you're dealing with a reference. In a lot of cases, this could be true.


python

comments powered by Disqus