Ruslan’s Blog

The other day I had to explain why in Python in contrast to Perl/Lisp/Scheme one can’t rebind variable outside of the local scope besides the global scope, i.e. the following function will result in an UnboundLocalError exception in Python 2.x

def outer():
    count = 0
    def inner():
        count += 1 # rebinding
    inner()
    print count

>>> outer()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "scope.py", line 8, in outer
    inner()
  File "scope.py", line 6, in inner
    count += 1 # rebinding
UnboundLocalError: local variable 'count' referenced before assignment

There are several workarounds.

For one thing it is possible to use a variable holding a mutable object:

def outer():
    ns = dict(count=0)
    def inner():
        ns['count'] += 1 # rebinding
    inner()
    print ns['count']

>>> outer()
1

Another approach is to use a function object itself to store the variable:

def outer():
    outer.count = 0
    def inner():
        outer.count += 1 # rebinding
    inner()
    print outer.count

>>> outer()
1

The good news is that there is no need for workarounds anymore in Python 3 which introduced nonlocal keyword:

def outer():
    count = 0
    def inner():
        nonlocal count
        count += 1 # rebinding
    inner()
    print(count)

>>> outer()
1

The other day I had to decode strings containing hex numbers into plain integers.

I could use built-in int([ x [, radix]]) with radix=16 to convert a string to an integer and forget about it, but I needed to decode pairs from the hex number so that in the end from the string ‘FFEEDD’ I get 255, 238, 221

But wouldn’t you know it, there is a standard solution to get pairs in Python and it’s built-in zip function.

It’s a well known function for experienced developers, but there is one documented feature of it that doesn’t always catch one’s eye and it’s a general form for clustering a data series into n-length groups zip(*[iter(s)]*n) which is described in official docs for zip built-in.

With that in mind solution to my problem boils down to a one-line decoder:

>>> s = 'FFDDEE'
>>> [int(''.join(pair), 16) for pair in zip(*[iter(s)]*2)]
[255, 221, 238]

For those wondering how it works:

1) iter(s) creates iterable from the s

>>> iter(s)
<iterator object at 0x8a3c2cc>

2) [iter(s)] * 2 creates list of 2 iterators that all use the same underlying iterable iter(s). The important thing is that created iterators share the same iterable, so when some data is consumed by first iterator that data becomes unavailable for the second iterator and vice versa.

You can see from the output that iterator objects are located at the same address. If you are new to this behavior you can read on about shallow copies.

>>> [iter(s)]*2
[<iterator object at 0x8a3c2cc>, <iterator object at 0x8a3c2cc>]

3) leading * in zip’s parameter unpacks list from step (2) into two separate parameters for zip function. Basically it becomes zip(iterator, iterator).

More about unpacking argument list you can read in official docs.

>>> zip(*[iter(s)]*2)
[('F', 'F'), ('D', 'D'), ('E', 'E')]

Sometimes you may need to have nested dictionaries initialized without much hassle.

“Autovivication” comes in handy. What you want to achieve is:

>>> d = dict()
>>> d['q']['w']['r'] = 777

Of course what you’ll get will be:

>>> d['q']['w']['r'] = 777
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'q'

If you don’t need deeply nested dictionaries defaultdict with dict default factory will do the job:

>>> from collections import defaultdict
>>> d = defaultdict(dict)
>>> d
defaultdict(<type 'dict'>, {})
>>> d['a']['b'] = 33
>>> d
defaultdict(<type 'dict'>, {'a': {'b': 33}})
>>> d['a']['b']
33

Here are another ways to have unlimited level of nested dictionaries.
1) Define custom class inheriting from built-in dict:

class default_dict(dict):
    def __missing__(self, key):
        self[key] = value = self.__class__()
        return value

>>> d = default_dict()
>>> d
{}
>>> d['q']['w']['r'] = 777
>>> d
{'q': {'w': {'r': 777}}}
>>> d['q']['w']['r']
777

2) Recursive function + defaultdict:

def make_dict():
    return defaultdict(make_dict)

>>> from collections import defaultdict
>>> d = defaultdict(make_dict)
>>> d
defaultdict(<function make_dict at 0xb7c5295c>, {})
>>> d['q']['w']['r'] = 777
>>> d
defaultdict(<function make_dict at 0xb7c5295c>, {'q': defaultdict(<function make_dict at 0xb7c5295c>, {'w': defaultdict(<function make_dict at 0xb7c5295c>, {'r': 777})})})
>>> d['q']['w']['r']
777

3) And for fun you can use Y Combinator (in short – making anonymous lambda recursive) which is actually as (2) but you don’t have to have recursive function defined:

def Y(g):
    return ((lambda f: f(f))
            (lambda f:
                 g(lambda x: f(f)
                   (x))))

defdict = Y(lambda mk_dict:
                lambda x=None: defaultdict(lambda x=None: mk_dict(x)))

>>> d = defdict()
>>> d
defaultdict(<function <lambda> at 0xb7c52a74>, {})
>>> d['q']['w']['r'] = 777
>>> d
defaultdict(<function <lambda> at 0xb7c52a74>, {'q': defaultdict(<function <lambda> at 0xb7c56c6c>, {'w': defaultdict(<function <lambda> at 0xb7c56a74>, {'r': 777})})})
>>> d['q']['w']['r']
777