A very common operation of Web servers these days is transferring files to clients. They do that by reading the files’ contents from a disk and writing it back to the clients’ sockets.

The copying of the file on Linux/UNIX, for example, could be done in a loop using read/write system calls:

import os
...
while True:
    data = os.read(filefd, 4096)
    if not data:
        break
    os.write(socketfd, data)

On the surface this looks perfectly fine, but for transferring large files, when pre-processing of the file contents isn’t necessary, this is pretty inefficient.

The reason is that the read and write system calls involve copying data from kernel space to user space and vice versa and all that happens in a loop:

That’s where the sendfile system call comes in handy. It provides a nice optimization for this particular use case by doing all the copying from the file descriptor to the socket descriptor completely in the kernel space:

Python 3.3 provides os.sendfile as part of the standard library. Check out more details on the GitHub.

Imagine that your server process needs to simultaneously wait for I/O on multiple descriptors and also needs to wait for the delivery of a signal.
The problem is that you can’t safely mix select() with signals because there is a chance of race condition.

Consider the following code excerpt:

GOT_SIGNAL = False

def handler(signum, frame):
    global GOT_SIGNAL
    GOT_SIGNAL = True

...

signal.signal(signal.SIGUSR1, handler)

# What if the signal arrives at this point?

try:
    reads, writes, excs = select.select(rlist, wlist, elist)
except select.error as e:
    code, msg = e.args
    if code == errno.EINTR:
        if GOT_SIGNAL:
            print 'Got signal'
    else:
        raise

The problem here is that if the SIGUSR1 is delivered after setting the signal handler but before call to select then the select call will block and we won’t execute our application logic in response to the event thus effectively “missing” the signal (our application logic in this case is printing the message: Got signal).

That’s an example of possible nasty racing. Let’s simulate that with selsigrace.py

Start the program

$ python selsigrace.py
PID: 32324
Sleep for 10 secs

and send the USR1 signal to the PID(it’s different on every run) within the 10 second interval while the process is still sleeping:

$ kill -USR1 32324

You should see the program produce additional line of output ‘Wake up and block in “select”‘ and block without exiting, no message “Got signal”:

$ python selsigrace.py
PID: 32324
Sleep for 10 secs
Wake up and block in "select"

If you send yet another USR1 signal at this point then the select will be interrupted and the program will terminate with the message:

$ kill -USR1 32324

$ python selsigrace.py
PID: 32324
Sleep for 10 secs
Wake up and block in "select"
Got signal

Self-Pipe Trick is used to avoid race conditions when waiting for signals and calling select on a set of descriptors.

The following steps describe how to implement it:

1. Create a pipe and change its read and write ends to be nonblocking
2. Add the read end of the pipe to the read list of descriptors given to select
3. Install a signal handler for the signal we’re concerned with. When the signal arrives the signal handler writes a byte of data to the pipe. Because the write end of the pipe is nonblocking we prevent the situation when signals flood the process, the pipe becomes full and the process blocks itself in the signal handler.
4. When select successfully returns check if the read end of the pipe is in the readables list and if it is then our signal has arrived.
5. When the signal arrives read all bytes that are in the pipe and execute any actions that have to be done in response to the signal delivery.

You can check out the implementation on GitHub and try it.

Start the selfpipe.py and send a USR1 signal to it. You should see it output a message Got signal and exit.

Have you ever seen a socket.error: [Errno 32] Broken pipe message when running a Python Web server and wondered what that means?

The rule is that when a process tries to write to a socket that has already received an RST packet, the SIGPIPE signal is sent to that process which causes the Broken pipe socket.error exception.

Here are two scenarios that you can try that cause SIGPIPE signal to be fired.

1. Server may send an RST packet to a client to abort the socket connection but the client ignores the packet and continues to write to the socket.

To test that behavior install Cynic, run it

$ cynic
INFO     [2012-06-08 05:06:37,040] server: Starting 'HTTPHtmlResponse'   on port 2000
INFO     [2012-06-08 05:06:37,040] server: Starting 'HTTPJsonResponse'   on port 2001
INFO     [2012-06-08 05:06:37,040] server: Starting 'HTTPNoBodyResponse' on port 2002
INFO     [2012-06-08 05:06:37,040] server: Starting 'HTTPSlowResponse'   on port 2003
INFO     [2012-06-08 05:06:37,040] server: Starting 'RSTResponse'        on port 2020
INFO     [2012-06-08 05:06:37,040] server: Starting 'RandomDataResponse' on port 2021
INFO     [2012-06-08 05:06:37,040] server: Starting 'NoResponse'         on port 2022
INFO     [2012-06-08 05:06:37,041] server: Starting 'LogRecordHandler'   on port /tmp/_cynic.sock

and then run the client1.py:

import socket

# connect to Cynic's RSTResponse service on port 2020
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('', 2020))

# first read gets an RST packet
try:
    s.recv(1024)
except socket.error as e:
    print e
    print

# write after getting the RST causes SIGPIPE signal
# to be sent to this process which causes a socket.error
# exception
s.send('hello')

and see what happens:

$ python ./client1.py
[Errno 104] Connection reset by peer

Traceback (most recent call last):
  File "./client1.py", line 17, in
    s.send('hello')
socket.error: [Errno 32] Broken pipe

2. Server can send an RST to the client’s SYN request to indicate that there is no process wating for connections on the host at the specified port, but the client tries to write to the socket anyway.

To test it run the client2.py:

import socket

# connect to the port that nobody is listening on
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# gets us an RST packet
try:
    s.connect(('', 30301))
except socket.error as e:
    print e
    print

# write after getting RST causes SIGPIPE signal
# to be sent to this process which causes an exception
s.send('hello')

Here is the output:

$ python ./client2.py
[Errno 111] Connection refused

Traceback (most recent call last):
  File "./sigpipe2.py", line 15, in
    s.send('hello')
socket.error: [Errno 32] Broken pipe

I hope that clarifies a SIGPIPE’s nature a little bit.

I’ve created a repo on GitHub called csdesign – https://github.com/rspivak/csdesign  – where I put several simple examples of concurrent TCP servers in Python.

Currently it contains two of those:

1. TCP Concurrent Server, One Child per Client (fork)
2. TCP Concurrent Server, I/O Multiplexing (select)

It’s an ongoing effort and if it’s something that you’re interested in just follow the project on GitHub or its progress on Twitter.

Here is the Roadmap:

• TCP Concurrent Server, One Thread per Client
• TCP Concurrent Server, I/O Multiplexing (poll)
• TCP Concurrent Server, I/O Multiplexing (epoll)
• TCP Preforked Server
• TCP Prethreaded Server

These days almost any application has several integration points like database, payment gateway, or some Web service that it consumes over HTTP.

All communication with the remote systems happens over the network and both networks and those systems often go wonky.

If we do not test the behavior of our system when the remote end operates out of spec and goes haywire the only place for testing becomes in production which is, as we all know, for some systems is less than acceptable.

Because the calls to the remote systems use network, the socket connection can have different failure scenarios, for example:

• The remote end resets the connection by sending a TCP RST packet
• The connection may be established, but the response is never sent back and the connection is not closed (If you don’t use socket timeouts in your app you may be in trouble at some point).
• The remote end can send garbage data as the response
• The service can send HTML over HTTP instead of the expected JSON response
• The HTTP service can send one byte of the response data every 30 seconds
• The remote HTTP service sends only headers and no body
• The service can send megabytes of data instead of expected kilobytes
• Etc.

It would be good to be able to test the behavior of our application when some of those conditions happen.

Cynic tries to help with that testing. Basically it’s a test harness (test double) that can be used to simulate crafty and devious remote systems right from your command line.

Cynic will try hard to cause injury to your system.

It’s goal is to make your system under test cynical.

Installation

$ [sudo] pip install cynic

Or the bleeding edge version from the git master branch:

$ [sudo] pip install git+https://github.com/rspivak/cynic.git#egg=cynic

Quick intro

Start Cynic which in turn will start multiple services on different ports:

$ cynic
INFO     [2012-06-03 23:44:35,603] server: Starting 'HTTPHtmlResponse'   on port 2000
INFO     [2012-06-03 23:44:35,603] server: Starting 'HTTPJsonResponse'   on port 2001
INFO     [2012-06-03 23:44:35,604] server: Starting 'HTTPNoBodyResponse' on port 2002
INFO     [2012-06-03 23:44:35,604] server: Starting 'HTTPSlowResponse'   on port 2003
INFO     [2012-06-03 23:44:35,604] server: Starting 'RSTResponse'        on port 2020
INFO     [2012-06-03 23:44:35,604] server: Starting 'RandomDataResponse' on port 2021
INFO     [2012-06-03 23:44:35,604] server: Starting 'NoResponse'         on port 2022
INFO     [2012-06-03 23:44:35,604] server: Starting 'LogRecordHandler'   on port /tmp/_cynic.sock

Test different services

1. Connect to the service on port 2020 and get a TCP RST packet right away which causes ‘Connection reset by peer’ message on the command line

$ curl http://localhost:2020
curl: (56) Recv failure: Connection reset by peer

2. Connect to the service on port 2021 and get back 7 bytes of random data

$ telnet localhost 2021
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
#6
Connection closed by foreign host.

3. Connect to the service on port 2001 to get the HTTP JSON response

$ curl http://localhost:2001
{"message": "Hello, World!"}

So you got the basic idea. Point your application’s integration points to the Cynic’s services and see how your app reacts.

For more information check the project’s GitHub page at https://github.com/rspivak/cynic

UPDATE [2012-06-04] Added screencast

Good old log files are still the most reliable, versatile, and useful sources of information.
When they are also human-readable and easy to use with standard Unix tools like tail and grep they are even more useful.

We all know that especially in time of stress when things with a system are going south having a quick and easy way to grep for useful information is critical. The purpose of a small Python utility Logsna is to provide a sane log output format that makes some grepping a bit easier.

Logsna offers a custom formatter class logsna.Formatter that can be used in a logging config file, for example:

# sanefmt.py
import logging
import logging.config
from StringIO import StringIO

CONFIG = """\
[loggers]
keys=root

[handlers]
keys=console

[handler_console]
class=logging.StreamHandler
args=(sys.stderr,)
formatter=sane

[formatters]
keys=sane

[logger_root]
level=DEBUG
handlers=console

# Our custom formatter class
[formatter_sane]
class=logsna.Formatter
"""

config = StringIO(CONFIG)
logging.config.fileConfig(config)

log = logging.getLogger('mylogger.component1')

log.debug('debug message')
log.info('info message')
log.warning('warning message')
log.critical('critical message')
try:
    1 / 0
except:
    log.exception('Houston we have a problem')

The Log Format

Here is an output from the above program:

DEBUG    [2012-05-21 01:59:23,686] mylogger.component1: debug message
INFO     [2012-05-21 01:59:23,686] mylogger.component1: info message
WARNING  [2012-05-21 01:59:23,686] mylogger.component1: warning message
CRITICAL [2012-05-21 01:59:23,686] mylogger.component1: critical message
ERROR    [2012-05-21 01:59:23,686] mylogger.component1: Houston we have a problem
! Traceback (most recent call last):
!   File "/home/alienoid/python/sanefmt.py", line 67, in
!     1 / 0
! ZeroDivisionError: integer division or modulo by zero

The Log Format Notes

- All timestamps are in ISO8601 and UTC format

- To grep for messages of a specific level

$ tail -f sanefmt.log | grep '^INFO'

- To grep for messages from a particular logger

$ tail -f sanefmt.log | grep 'component1:'

- To pull out full exception tracebacks with a corresponding log message

$ tail -f sanefmt.log | grep -B 1 '^\!'

Installation

$ [sudo] pip install logsna

I’ve recently discovered a neat BASH specific syntax for stream redirection so I thought I’d share it on the off chance you find it useful too.

Let’s see how standard error (STDERR) can be redirected to standard output (STDOUT), which in turn is redirected to a file, using standard syntax:

$ printf "%s\n%\n" Hello world > FILE 2>&1
$ cat FILE
Hello
bash: printf: `\': invalid format character

The order is important:

$ printf "%s\n%\n" Hello world 2>&1 > FILE
bash: printf: `\': invalid format character
$ cat FILE
Hello

And here is a BASH nonstandard syntax for redirection of STDERR and STDOUT to the same place – &> FILE

$ printf "%s\n%\n" Hello world &> FILE
$ cat FILE
Hello
bash: printf: `\': invalid format character

I made some changes to the lexer and now SlimiIt can handle JavaScript string literals that span multiple lines.
In JavaScript, as in Python, continuation lines can be used with a backslash as the last character on the line indicating that the next line is a continuation of the line.

Here is a simple example:

$ cat test.js
var s = "hello \
    \
    world\
    ";
$
$ cat test.js | slimit -m -t
var a="hello         world    ";

While the new version of SlimIt is mostly a bugfix release:
- Problem with “+”, “++” and whitespace
- Problem with for loop with ? operator in initializer position

there is also one new feature: -t / –mangle-toplevel command-line option.
It allows you to turn on global scope name mangling which is now off by default.
Let’s see it in action.

$ cat test.js
function foo(first, second){
  alert(first, second);
}

foo();

Global scope mangling is off by default:

$ slimit -m test.js
function foo(a,b){alert(a,b);}foo();

With global scope name mangling on:

$ slimit -m --mangle-toplevel test.js
function a(a,b){alert(a,b);}a();

In Python code pass mangle_toplevel=True argument to minify function:

>>> from slimit import minify
>>> text = """\
... function foo(first, second){
...   alert(first, second);
... }
...
... foo();
... """
>>> minify(text, mangle=True, mangle_toplevel=True)
'function a(a,b){alert(a,b);}a();'

Crammit 0.3 is out.

Two main features in this release:
- Python 2.6 support (thanks to Didip Kerabat for contributing code)
- files option in assetsinfo.yaml that lists all processed files for a bundle

After running it on the command line

$ crammit -c assets.yaml

the output information from assetsinfo.yaml may look like

css:
  base:
    files:
    - /home/rspivak/static/css/test1.css
    - /home/rspivak/static/css/test2.css
    fingerprint: 71fe4cba05a1a51023c6af4c4abf9c47ab21e357
    output:
      gz: base-71fe4cba05a1a51023c6af4c4abf9c47ab21e357.min.css.gz
      min: base-71fe4cba05a1a51023c6af4c4abf9c47ab21e357.min.css
      raw: base-71fe4cba05a1a51023c6af4c4abf9c47ab21e357.css
    size:
      gz: 108
      min: 235
      raw: 277
javascript:
  common:
    files:
    - /home/rspivak/static/js/application.js
    - /home/rspivak/static/js/vendor/vendor1.js
    - /home/rspivak/static/js/vendor/vendor2.js
    fingerprint: 6493b619c73c49ce1f4dfe2c31d41902e98acaee
    output:
      gz: common-6493b619c73c49ce1f4dfe2c31d41902e98acaee.min.js.gz
      min: common-6493b619c73c49ce1f4dfe2c31d41902e98acaee.min.js
      raw: common-6493b619c73c49ce1f4dfe2c31d41902e98acaee.js
    size:
      gz: 56
      min: 41
      raw: 50

It’s very easy to add a class-based view to your Pyramid application.

There are basically just two things that you have to do with your typical Python class:

1. make sure that your __init__ method accepts and stores request object for further reference (request is not passed to your other class methods)
2. add a @view_config decorator to a method that you want to be called when your view is rendered

Here is a simple example of a login class-based view:

from pyramid.view import view_config
from pyramid.httpexceptions import HTTPForbidden

class LoginView(object):
    def __init__(self, request):
        self.request = request

    def get_login_url(self):
        login_url = self.request.route_url('login')
        return login_url

    @view_config(context=HTTPForbidden, renderer='login.mak')
    @view_config(route_name='login', renderer='login.mak')
    def login(self):
        login_url = self.get_login_url()
        # more code goes here
        return dict(url=login_url)

As you can see it really is that simple.

You’ve probably noticed that I had to repeat renderer=’login.mak’ part twice and it’s kind of annoying. But as luck would have it as of version 1.3a2 Pyramid comes with a new decorator @view_defaults and our class can be re-written with it like this:

from pyramid.view import view_config, view_defaults
from pyramid.httpexceptions import HTTPForbidden

@view_defaults(renderer='login.mak')
class LoginView(object):
    def __init__(self, request):
        self.request = request

    def get_login_url(self):
        login_url = self.request.route_url('login')
        return login_url

    @view_config(context=HTTPForbidden)
    @view_config(route_name='login')
    def login(self):
        login_url = self.get_login_url()
        # more code goes here
        return dict(url=login_url)

I like that much more than the old version. For more information on the @view_defaults decorator see here: @view_defaults decorator

And the last thing I’d like to mention is a handy view variable that is available for you in your templates. You can use it to call helper methods from your class-based view like this:

<span>Login URL: ${view.get_login_url()}</span>

UPDATE Mar 2, 2012

If you don’t want to use the view_config decorator and want to register your class-based view’s method via an add_view call of your Configurator object this is how to do it.

# __init__.py
from pyramid.config import Configurator

from mypackage.views import LoginView

def main(global_config, **settings):
    config = Configurator(settings=settings)
    config.add_route('login', '/login')
    config.add_view(LoginView, attr='login', route_name='login')
    ...

You pass your view class (it can also be a Python dotted name) to the add_view method and use an attr argument to specify what method on that class should be used as a view.

Pyramid 1.3 introduced different p* scripts that replaced the paster commands with Pyramid-specific analogues. Here is a summary table that lists all currently available scripts with their brief descriptions.

The best way to get to know what those commands do is to try them out. But before we see our p* scripts in action make sure to install the latest Pyramid 1.3 and activate your virtual environment. The steps to do that might look like this:

$ virtualenv --no-site-packages test
$ cd test
$ source bin/activate
$ pip install pyramid

OK, buckle up and let’s get it started.

Create application

Let’s create a simple project called myproject with pcreate command

$ pcreate -s starter myproject
...
Welcome to Pyramid.  Sorry for the convenience.

Run it

In order to run our application we need to install it for development first.

$ cd myproject
$ python setup.py develop

And now we can run it with pserve script

$ pserve development.ini
Starting server in PID 17410.
serving on http://0.0.0.0:6543

Display all application routes

$ proutes development.ini | grep -v debug
Name        Pattern             View
----        -------             ----
__static/   /static/*subpath    <function <pyramid.static  ...>
home        /                   <function my_view at 0x996b09c>

Display matching views for a given URL

$ pviews development.ini /

URL = /

    context: <pyramid.traversal.DefaultRootFactory instance at ...>
    view name: 

    Route:
    ------
    route name: home
    route pattern: /
    route path: /
    subpath: 

        View:
        -----
        myproject.views.my_view

Send a request to our application without starting a server

Let’s make a GET request and show response headers

$ prequest -d development.ini /
200 OK
Content-Type: text/html; charset=UTF-8
Content-Length: 3390
<!DOCTYPE html PUBLIC ...>
...

And now a POST request to a non-existent URL

$ echo 'body text' | prequest -mPOST -d development.ini /dummy
404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 158
<html>
 <head>
  <title>404 Not Found</title>
 </head>
 <body>
  <h1>404 Not Found</h1>
  The resource could not be found.<br/><br/>
/dummy

 </body>
</html>

Play in the interactive shell

$ pshell development.ini
Python 2.7.1 (r271:86832, Apr 12 2011, 16:16:18)
[GCC 4.6.0 20110331 (Red Hat 4.6.0-2)] on linux2
Type "help" for more information.

Environment:
  app          The WSGI application.
  registry     Active Pyramid registry.
  request      Active request object.
  root         Root of the default resource tree.
  root_factory Default root factory used to create `root`.

>>> from myproject.views import my_view
>>> from pyramid.request import Request
>>> r = Request.blank('/')
>>> my_view(r)
{'project': 'myproject'}

If we install IPython it will be used instead of the default Python shell.

Display “tweens” (middleware)

$ ptweens development.ini
...

Implicit Tween Chain

Position    Name
--------    ----
-           INGRESS
0           pyramid_debugtoolbar.toolbar_tween_factory
1           pyramid.tweens.excview_tween_factory
-           MAIN

If you’ve read this far and want to dig deeper into those commands check out an excellent official documentation page: Command-Line Pyramid

Crammit is really simple. It’s a small tool that uses YAML for configuration and other libraries to do the heavy lifting.

Here is a config file assets.yaml:

output: assets    # directory path relative to  current directory
fingerprint: true # add sha1 hash to the output file name

javascript:
  # 'common' is a bundle name
  # output file will have prefix 'common'
  common:
    - static/js/application.js
    - static/js/vendor/*.js
  utils:
    # paths are relative to the current directory
    - static/js/utils.js

css:
  base:
    # you can use Unix shell-style wildcards in file names
    - static/css/*.css

Let’s put it through its paces:

$ crammit -c assets.yaml

In addition to generated asset bundles it outputs an information file that sums up all what’s been done.

css:
  base:
    fingerprint: 71fe4cba05a1a51023c6af4c4abf9c47ab21e357
    output:
      gz: base-71fe4cba05a1a51023c6af4c4abf9c47ab21e357.min.css.gz
      min: base-71fe4cba05a1a51023c6af4c4abf9c47ab21e357.min.css
      raw: base-71fe4cba05a1a51023c6af4c4abf9c47ab21e357.css
    size:
      gz: 108
      min: 235
      raw: 277
javascript:
  common:
    fingerprint: 6493b619c73c49ce1f4dfe2c31d41902e98acaee
    output:
      gz: common-6493b619c73c49ce1f4dfe2c31d41902e98acaee.min.js.gz
      min: common-6493b619c73c49ce1f4dfe2c31d41902e98acaee.min.js
      raw: common-6493b619c73c49ce1f4dfe2c31d41902e98acaee.js
    size:
      gz: 56
      min: 41
      raw: 50
  utils:
    fingerprint: c3ef63280b954d99e8b13fc11ea3031caee77f1a
    output:
      gz: utils-c3ef63280b954d99e8b13fc11ea3031caee77f1a.min.js.gz
      min: utils-c3ef63280b954d99e8b13fc11ea3031caee77f1a.min.js
      raw: utils-c3ef63280b954d99e8b13fc11ea3031caee77f1a.js
    size:
      gz: 42
      min: 22
      raw: 24

Check it out here for more information.

I know some people think that Pyramid is a complex framework and that aspiring Python web developers should start with something else and maybe come back to Pyramid later.

But I truly believe that even if you’re a total beginner Pyramid can serve you really well and once you grow more experienced, you’ll appreciate the power that you get with the Pyramid framework.

Take a look at this “hello world” application in hello.py:

from wsgiref.simple_server import make_server
from pyramid.config import Configurator

def hello(request):
   return 'Hello %(name)s!' % request.matchdict

if __name__ == '__main__':
    config = Configurator()
    config.add_route('main', '/hello/{name}')
    config.add_view(hello, route_name='main', renderer='string')
    server = make_server('', 8080, config.make_wsgi_app())
    server.serve_forever()

It wasn’t that difficult.

And this is how to install and run it:

$ pip install pyramid
$ python hello.py

Then in your browser type http://localhost:8080/hello/fred and see the result. Easy peasy.

So what are you waiting for? Grab it here and give it a try!

Let’s compare compression of SlimIt and YUI Compressor using jQuery source file.

As you can see SlimIt performs better and it’s pure Python.
Do you still use YUI Compressor in your Python project? Give SlimIt a try.
If you have any questions or encountered any issue here is the place to discuss it: GitHub Issues

First of all Happy New Year!

I moved my blog to a new hosting provider and as you can see the blog is somewhat empty.
Sorry for the inconveniences. I’ll be reviewing my old content and populating my new blog with it over time.

New release of httpcode is out. Grab it here.

This release adds colorization of HTTP codes:

SlimIt got a new feature – object access minification.

Let’s give it a try:

>>> from slimit import minify
>>> minify('foo["bar"];') == 'foo.bar;'
True

But if the object attribute is not a legitimate identifier minification does nothing:

>>> minify('foo["bar bar"];')
'foo["bar bar"];'

This is not a part of an official release yet, but you can clone it and install from GitHub.
Let me know if you have any issues with it.

A quick announcement – httcode 0.4 is out. Grab it while it’s hot!

Thanks to Mark Striemer in this release you can filter codes with regex and use ‘x‘ to denote any digit.

Filter codes with a regex

$ hc 30[12]
Status code 301
Message: Moved Permanently
Code explanation: Object moved permanently -- see URI list

Status code 302
Message: Found
Code explanation: Object moved temporarily -- see URI list

Use an ‘x’ for any digit

$ hc 1xx
Status code 100
Message: Continue
Code explanation: Request received, please continue

Status code 101
Message: Switching Protocols
Code explanation: Switching to new protocol; obey Upgrade header

Grab it here

In addition to command line version here comes Emacs package to explain HTTP status codes in minibuffer httpcode.el

Now and then I find myself looking at RFC or Wikipedia page to find the meaning of different HTTP status codes, like 100, 405, 410, etc. It takes some time to click on the link, wait for the page to load and to find the code I need.

That’s a pretty slow workflow so I wrote real quick a little utility called httpcode that explains the meaning of an HTTP status code on the command line. Who needs a browser anyways Try it.

Installation

$ [sudo] pip install httpcode

Examples

Explain 405 status code

$ hc 405
Status code 405
Message: Method Not Allowed
Code explanation: Specified method is invalid for this resource.

List all codes

$ hc
Status code 100
Message: Continue
Code explanation: Request received, please continue

Status code 101
Message: Switching Protocols
Code explanation: Switching to new protocol; obey Upgrade header

Status code 200
Message: OK
Code explanation: Request fulfilled, document follows

...

Show help

$ hc -h
Usage: hc 1

Without parameters lists all available
HTTP status codes and their description

Options:
  -h, --help  show this help message and exit

Check out the documentation here.

Quick update:

1. SlimIt is now available on Read the Docs

2. pip is now the only tool mentioned in installation documentation, no more easy_install. I also got a GitHub cake for that from Kenneth Reitz

I’ve added a base node visitor class ASTVisitor to SlimIt to reduce some boilerplate code required for writing a custom node visitor.

The class itself is pretty simple and contains just two methods:

• a visit method that you want to call to process all node’s children and
• a generic_visit method that acts as a fallback when your visitor doesn’t know how to visit a particular node

Let’s say that you want to parse a JavaScript object literal. In order to do that your visitor has to inherit from ASTVisitor and add a visit_Object method. One thing to note here is a signature of the method: def visit_NodeType(self, node) where NodeType is the type of a node you want to visit. For a full list of available node types check out ast.py module on GitHub

With that said the code to visit an object literal with our own node visitor now would look like this:

>>> from slimit.parser import Parser
>>> from slimit.visitors.nodevisitor import ASTVisitor
>>>
>>> text = """
... var x = {
...     "key1": "value1",
...     "key2": "value2"
... };
... """
>>>
>>> class MyVisitor(ASTVisitor):
...     def visit_Object(self, node):
...         """Visit object literal."""
...         for prop in node:
...             left, right = prop.left, prop.right
...             print 'Property key=%s, value=%s' % (left.value, right.value)
...             # visit all children in turn
...             self.visit(prop)
...
>>>
>>> parser = Parser()
>>> tree = parser.parse(text)
>>> visitor = MyVisitor()
>>> visitor.visit(tree)
Property key="key1", value="value1"
Property key="key2", value="value2"

I needed a simple server that could be used as a stub for testing Python SFTP clients so I whipped out one over the weekend.
It’s a simple single-threaded server so it’s not for production use but give it a try anyways, you might find it useful too.

Installation:

$ [sudo] pip install sftpserver

Running the server:

    $ sftpserver
    Usage: sftpserver [options]
    -k/--keyfile should be specified

    Options:
      -h, --help            show this help message and exit
      --host=HOST           listen on HOST [default: localhost]
      -p PORT, --port=PORT  listen on PORT [default: 3373]
      -l LEVEL, --level=LEVEL
                            Debug level: WARNING, INFO, DEBUG [default: INFO]
      -k FILE, --keyfile=FILE
                            Path to private key, for example /tmp/test_rsa.key

    $ sftpserver -k /tmp/test_rsa.key -l DEBUG

Connecting with a Python client to our server:

>>> import paramiko
>>> pkey = paramiko.RSAKey.from_private_key_file('/tmp/test_rsa.key')
>>> transport = paramiko.Transport(('localhost', 3373))
>>> transport.connect(username='admin', password='admin', pkey=pkey)
>>> sftp = paramiko.SFTPClient.from_transport(transport)
>>> sftp.listdir('.')
['loop.py', 'stub_sftp.py']

The server code itself is pretty minimal, check it out on GitHub.

I got a question about how to get to the values of object literal properties with SlimIt.
Here is one way to do it with a custom node visitor:

# parseobj.py
from slimit.parser import Parser

text = """
var x = {
    "key1": "value1",
    "key2": "value2"
};
"""

class MyVisitor(object):
    def visit(self, node):
        method = 'visit_%s' % node.__class__.__name__
        return getattr(self, method, self.generic_visit)(node)

    def generic_visit(self, node):
        for child in node:
            self.visit(child)

    def visit_Object(self, node):
        """Visit object literal."""
        for prop in node:
            left, right = prop.left, prop.right
            print 'Property value: %s' % right.value
            # visit all children in turn
            self.visit(prop)

parser = Parser()
tree = parser.parse(text)
visitor = MyVisitor()
visitor.visit(tree)

$ python parseobj.py
Property value: "value1"
Property value: "value2"

I’ve recently had to deal with parsing fixed-length file records and found struct.unpack and namedtuple to be a pretty useful and concise combination for the task:

>>> from struct import unpack
>>> from collections import namedtuple
>>> line = '    23C   17000'
>>> Transaction = namedtuple('Transaction', 'code status amount')
>>> format = '<6sc8s' # code: 6bytes, status: 1byte, amount: 8bytes
>>> txn = Transaction._make(unpack(format, line))
>>> txn
Transaction(code='    23', status='C', amount='   17000')
>>> txn.code, txn.status, txn.amount
('    23', 'C', '   17000')

I seem to get this question every half a year and every time I have difficulties remembering how to do it.

>>> from datetime import datetime
>>> d = datetime.utcnow()
>>> d
datetime.datetime(2011, 7, 21, 3, 13, 22, 259901)

So here is the snippet that I use to convert datetime to timestamp:

>>> import calendar
>>> calendar.timegm(d.utctimetuple())
1311218002

Let’s verify it:

$ date -ud @1311218002
Thu Jul 21 03:13:22 UTC 2011

Looks fine, but I can’t for the life of me understand why the function timegm is part of the calendar module.

From Python official documentation:

calendar.timegm(tuple)

An unrelated but handy function that takes a time tuple such as returned by the gmtime() function in the time module, and returns the corresponding Unix timestamp value, assuming an epoch of 1970, and the POSIX encoding. In fact, time.gmtime() and timegm() are each others’ inverse.

Have you ever had, at least once, an exception when logging an exception because you’d screwed up the number of logging arguments?

Something akin to the following:

>>> import logging
>>> msg1, msg2 = 'Look', 'leap!'
>>> try:
...     1 / 0
... except Exception:
...     # crash and burn - no msg2. where is my unit test?!!!
...     logging.exception('%s before you %s', msg1)
...
Traceback (most recent call last):
  ...
TypeError: not enough arguments for format string

Here is what you get in Erlang:

1> error_logger:info_msg("~s before you ~s~n", ["Look"]).
ok

=INFO REPORT==== 22-Jun-2011::23:09:49 ===
ERROR: "~s before you ~s~n" - ["Look"]
2>

That’s right – you get ‘ok’ and a report. That allows you to get critical information even if there is a problem with your log message arguments without blowing your whole system. I like that, makes the system more fault-tolerant.

It’s good to know a name for such a useful pattern, so here we go – The Bunch Pattern.

>>> class Bunch(dict):
...     def __init__(self, *args, **kw):
...         super(Bunch, self).__init__(*args, **kw)
...         self.__dict__ = self
...
>>> bunch = Bunch(name='Bunch', category='pattern')
>>> bunch.name
'Bunch'
>>> bunch['name']
'Bunch'
>>> 'pattern' in bunch
False
>>> bunch.pattern = True
>>> 'pattern' in bunch
True
>>> bunch
{'category': 'pattern', 'pattern': True, 'name': 'Bunch'}
>>>

When I worked on name mangling in SlimIt I needed a function that would generate subsets of a character string in lexicographic order.
For example, if I had a character sequence ‘abc’ I would expect to call my function and get items in the following order: ‘a’, ‘b’, ‘c’, ‘ab’, ‘ac’, ‘bc’, ‘abc’. The produced sequence is called a power set and binary counting is a well known algorithm for generating such power sets .

The idea is that we represent our character sequence as a binary string of n bits, then start counting from zero to 2**n – 1 and if a bit k is set to 1 then we would put k-th element from the character sequence into an output set.
Let’s say we have a sequence ‘abc’, then counting to 2**3 – 1 would produce binary numbers 000, 001, 010, 011, 100, …, which correspond to ”, ‘c’, ‘b’, ‘bc’, ‘a’, …, in the character sequence.

Code for binary counting looks like this (credit for original implementation goes to Tim Peters):

def powerset(s):
    """powerset('abc') -> a b ab c ac bc abc"""
    pairs = [(2**index, char) for index, char in enumerate(s)]
    for index in xrange(1, 2**len(s)):
        yield ''.join(char for mask, char in pairs if mask & index)

And while it generates all necessary subsets they are not in lexicographic order.

So I decided to look at itertools module and check closer some properties of the combinations function available there and see if it could fit for my purposes and – wouldn’t you know it? – at the very bottom of the module’s documentation page I found a powerset recipe that did just what I needed using combinations function.
Here is a little bit adapted version of it:

import itertools

def powerset(s):
    """powerset('abc') -> a b c ab ac bc abc"""
    for chars in itertools.chain.from_iterable(
        itertools.combinations(s, r) for r in range(1, len(s)+1)
        ):
        yield ''.join(chars)

I’ve been using Python for a long time now and still continue to bump into elegant solutions in Python standard library and that’s what I call the joy of Python

SlimIt 0.5 is out. This release includes name mangling that can be turned on with -m or –mangle flag from the command line.

To give you a taste of what it’s like here is a simple example:

function test() {
  var long_name = 'long name', log = 5;
  console.log(long_name);
  foo = function(arg1, arg2) {
    var arg2 = 'new arg2';
    console.log(long_name + arg1 + arg2);
  };
}

After name mangling:

function a() {
  var a = 'long name', b = 5;
  console.log(a);
  foo = function(b, c) {
    var c = 'new arg2';
    console.log(a + b + c);
  };
}

And here is a new jQuery 1.6.1 minification benchmark for jsmin, rJSmin, and SlimIt (the lower the better):

Please, file any bug reports at GitHub.

SlimIt 0.4 is released and it features block braces removal during minification process.

If a block statement contains only one source element then curly braces around that statement are removed by SlimIt:

with (obj) {
    a = b;
}

after minification will become:

with(obj)a=b;

And

if (true) {
    x = 1;
} else {
    x = 0;
}

will become

if(true)x=1;else x=0;

There is one case where braces removal gets tricky. Take a look at the following JavaScript code:

if ( obj ) {
    for ( n in obj ) {
        if ( v === false) {
            break;
        }
    }
} else {
    for ( ; i < l; ) {
        if ( nv === false ) {
            break;
        }
    }
}

If we blindly remove all block braces result will be the following (output is indented and formatted to illustrate the point):

if (obj)
    for (n in obj)
        if (v === false)
            break;
else
    for (; i < l; )
        if (nv === false)
            break;

but it will be interpreted by a JavaScript parser as

if (obj)
    for (n in obj)
        if (v === false)
            break;
        else
            for (; i < l; )
                if (nv === false)
                    break;

because else part will be associated with the closest if and that’s not what was described by the original pre-miniified source code. Because of that the minifier will convert such construct to the following form to avoid ambiguity but still performing minification where possible (indented for illustration purpose):

if (obj) {
    for (n in obj)
        if (v === false)
            break;
} else
    for (; i < l; )
        if (nv === false)
            break;

And this is how it’ll look like in minified form:

if(obj){for(n in obj)if(v===false)break;}else for(;i<l;)if(nv===false)break;')

Here are some new numbers for jsmin, rJSmin, and SlimIt minification of jQuery 1.6.1:

P.S: After minification by SlimIt I ran a test suite that comes with jQuery against the minified version and got the same results as with official minified version.

SlimIt is in its infancy yet but it can already minify some stuff out.

Here is a comparison of jsmin and SlimIt minification of jQuery 1.6:

As you can see SlimIt on this source file performs better reducing jquery-1.6.js by additional 964 bytes.

I have to note though that SlimIt is currently slower than jsmin and there are two contributing factors:
1. SlimIt is a complete source-to-source compiler where you have a parser. a lexer and the parser constructs an AST and then minified code is generated by an AST tree walker.
2. Minifier/tree walker uses += operator when concatenating strings and it hurts performance on large strings and multiple concatenations.
Standard solution is to use list.append() and ”.join() methods because they are faster, but I didn’t bother with the speed for now (that will be the target of next releases) – as they say “Make it work first, then make it faster”.

Due to the nature of the minifier (source-to-source compiler) there are many ways how to improve minification considerably, so stay tuned – next releases will give even more bang for the buck.

UPDATE: Looks like http://slimit.org doesn’t work reliably for everyone so here is a direct link to documentation where you’ll find all necessary information with further links: http://packages.python.org/slimit Meanwhile I’ll be looking for a place to host the package’s documentation reliably with slimit.org domain.

New release is out and SlimIt 0.3.2 (http://slimit.org) can now minify JavaScript code.

Let’s do it from the command line:

$ slimit -h
Usage: slimit [input file]

If no input file is provided STDIN is used by default.
Minified JavaScript code is printed to STDOUT.

$ cat test.js
var a = function( obj ) {
        for ( var name in obj ) {
                return false;
        }
        return true;
};
$
$ slimit < test.js
var a=function(obj){for(var name in obj){return false;}return true;};

And using library API:

>>> from slimit import minify
>>> text = """
... var a = function( obj ) {
...         for ( var name in obj ) {
...                 return false;
...         }
...         return true;
... };
... """
>>> print minify(text)
var a=function(obj){for(var name in obj){return false;}return true;};

If you find any issues, please, submit them at GitHub.

Happy minifying

SlimIt 0.2 (should be also accessible via http://slimit.org) got a JavaScript parser, pretty printer and a node visitor.

Let’s iterate over, modify a JavaScript AST and pretty print it:

>>> from slimit.parser import Parser
>>> from slimit.visitors import nodevisitor
>>> from slimit import ast
>>>
>>> parser = Parser()
>>> tree = parser.parse('for(var i=0; i<10; i++) {var x=5+i;}')
>>> for node in nodevisitor.visit(tree):
...     if isinstance(node, ast.Identifier) and node.value == 'i':
...         node.value = 'hello'
...
>>> print tree.to_ecma() # print awesome javascript 
for (var hello = 0; hello < 10; hello++) {
  var x = 5 + hello;
}
>>>

SlimIt comes with a bunch of tests and I also used it to parse and pretty print jQuery 1.5.2 which I then tested in a browser and it worked fine. But, you know, software is not perfect If you find any issues, please, submit them at GitHub.

Enjoy.

SlimIt 0.1 is out. This release contains only a JavaScript lexer and doesn’t minify anything yet.

Why to release when it doesn’t minify yet?

The foundation of the source-to-source compiler is a working lexer, so I decided to release it early on the off chance interested people would test it and report any issues with it while I work on parser and translator parts and some might even find the lexer useful on its own.

To install SlimIt just run:

$ sudo pip install slimit

Then you can use it in your programs like this:

>>> from slimit.lexer import Lexer
>>> lexer = Lexer()
>>> lexer.input('a = 1;')
>>> for token in lexer:
...     print token
...
LexToken(ID,'a',1,0)
LexToken(EQ,'=',1,2)
LexToken(NUMBER,'1',1,4)
LexToken(SEMI,';',1,5)

For more information and examples visit http://slimit.org

Here are two new things I learned about JavaScript when working on this release:

1. Identifiers can contain unicode escape sequences but not all of them are allowed (for allowed unicode categories check out section 7.6 of ECMA-262 specification).

This one is valid:

 var u03c0_tail = 7;

But this one is not:

 var u0036_tail = 3;

Per specification (section 7.6 of ECMA-262):

"A UnicodeEscapeSequence cannot be used to put a character into an IdentifierName that would
 otherwise be illegal. In other words, if a  UnicodeEscapeSequence sequence were replaced by its
 UnicodeEscapeSequence's CV, the result must still be valid IdentifierName that has the exact
 same sequence of characters as the original IdentifierName."

That rule basically explains why u0036_tail is not a legal identifier. It happens because character value (CV) of u0036 is DIGIT SIX and 6_tail is not a valid identifier because an identifier can’t start with a digit.

2. Ambiguity at a lexical level between /, /= and regex literal.
This case has to be handled separately in the code by keeping track of a previous token, bummer.

Here are two examples from Mozilla site that demonstrate the problem at hand:

for (var x = a in foo && "</x>" || mot ? z:/x:3;x<5;y</g/i) {xyz(x++);}
for (var x = a in foo && "</x>" || mot ? z/x:3;x<5;y</g/i) {xyz(x++);}

In the first example /x:3;x<5;y</g/i is a regex literal, in the second it’s /g/i because z/x is interpreted as division.

I’d like to thank Ned Batchelder for releasing code of jslex. I used many test cases and some regex matchers from that project.

UPDATE: Some people experienced problems when accessing http://slimit.org – redirection didn’t work properly. Check out this page if you had that issue.

Sometimes I need to indent a block of text in Emacs regardless of current major mode. A key binding that fits the bill is C-x TAB which runs the command indent-rigidly.

If used with prefix argument C-u it will indent by specified number of spaces, so pressing C-u 8 C-x TAB after marking a block of text will indent that block by 8 spaces and C-u -8 C-x TAB will remove 8 spaces of indentation. There is a nice shortcut: C-u C-x TAB will indent a block of text by 4 spaces.

When in Python mode marking a region and pressing C-c > to shift lines in region to the right and C-c < to shift lines in region to the left works perfectly well too.

I’ve been using this small Python gem for couple years now and it really comes in handy when you need a quick web server running without setting up nginx or apache or what have you.

Just open up a terminal and type:

$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...

Now check it in your browser: http://localhost:8000/

You can also change port the server is listening on:

$ python -m SimpleHTTPServer 7070
Serving HTTP on 0.0.0.0 port 7070 ...

After finishing up initial work for the TinyPie interpreter and AST visualizer I’m well on my way to TinyPie compiler.

For simplicity reasons and to better understand different parts involved I decided to go with Register-Based Bytecode Interpreter / Virtual Machine , like the one used by Lua and Dalvik, instead of directly generating binary code for target architecture. The main source of inspiration, ideas, and examples were the book Language Implementation Patterns Ch.10(Building Bytecode Interpreters) and the paper The Implementation of Lua 5.0

Here is a quick overview of developed components:

1. Bytecode Assembler
2. Register-Based Virtual Machine

Bytecode Assembler converts assembly program into binary bytecodes. The bytecode is further interpreted by the TinyPie Register-Based Bytecode Interpreter / Virtual Machine.

TinyPie Assembly language grammar:

Assembler yields the following components:

1. Code memory: This is a bytearray containing bytecode instructions and their operands derived from the assembly source code.
2. Global data memory size: The number of slots allocated in global memory for use with GSTORE and GLOAD assembly commands.
3. Program entry point: An address of main function .def main: ...
4. Constant pool: A list of objects (integers, strings, function symbols) that are not part of the code memory. Bytecode instructions refer to those objects via integer index.

Here is a factorial function in TinyPie language:

Here is an equivalent TinyPie assembly code:

Here are the resulting elements produced by the Bytecode Assembler after translating the above assembly code:

Virtual Machine architecture(Virtual Machine simulates a hardware processor with general-purpose registers.):

1. IP: Instruction pointer register that points into the code memory at the next instruction to execute.
2. CPU: Instruction dispatcher that simulates fetch-decode-execute cycle with a switch (if elif) statement in a loop – reads bytecode at IP, decodes its operands and executes corresponding operation.
3. Global data memory: a list of global objects. Contents is accessed via integer index.
4. Code memory: Holds bytecode instructions and their operands.
5. Call stack: Holds StackFrame objects with function return address, parameters, and local variables.
6. Stack frame: A StackFrame object that holds all required information to invoke a function:
   • function symbol
   • function return address
   • registers hold return value, arguments, locals, and temporary values
7. FP: Frame pointer – a special-purpose register that points to the top of the function call stack
8. Constant pool: Integers, strings, and function symbols all go into constant pool. Instructions refer to constant pool values via an integer index.

High-level overview:

Bytecode instructions for TinyPie VM:

TinyPie VM comes with a tpvm command line utility:

Sample program:

Running VM with the sample program as an input:

The missing piece is conversion from TinyPie AST to TinyPie Assembly code that I need to implement for the TinyPie compiler.

If you want to see how TinyPie AST (Abstract Syntax Tree) for different language constructs looks like you can now use gendot command line utility that I’ve added to the project and that is available in the development mode when you use buildout. This utility generates DOT file than can be further processed by dot program to draw nice graphs.

This is how it works.
First you need to clone or git pull changes from the tinypie repo.

$ cd tinypie
$ git pull

Rerun buildout to generate gendot utility.

$ bin/buildout

Now you can generate DOT output for TinyPie AST. You can pass a file name to gendot or send some text to its standard input.
To generate DOT file for function call, for example, do the following:

$ echo -n 'foo(3)' | bin/gendot
digraph astgraph {
   node [shape=plaintext, fontsize=12, fontname="Courier", height=.1];
   ranksep=.3;
   edge [arrowsize=.5]

   node1 [label="BLOCK"];
   node2 [label="CALL"];
   node3 [label="ID (foo)"];
   node4 [label="INT (3)"];

   node2 -> node3
   node2 -> node4
   node1 -> node2
}

If you save that output and process it further with the dot command you’ll get a nice image of function call AST:

$ echo -n 'foo(3)' | bin/gendot > funcall.dot
$ dot -Tpng -o funcall.png funcall.dot
$ ls -l funcall.png
-rw-rw-r--. 1 alienoid alienoid 4617 Jan 23 17:37 funcall.png

This is how AST looks like for a factorial program:

$ cat factorial.tp
print factorial(5)

def factorial(x):
  if x < 2 return 1
  return x * factorial(x - 1)
.
$ cat factorial.tp | bin/gendot > factorial.dot
$ dot -Tpng -o factorial.png factorial.dot

I’ve been working recently on a tree-based interpreter for a very simple Python-like programming language called TinyPie.
In contrast to Syntax-Directed Intrepreter this one executes source code by constructing Abstract Syntax Tree from homogeneous nodes and walking the tree.

The language is based on Pie language from Language Implementation Patterns Ch.9

To install the interpreter use either pip install (alternatively you can run $ sudo easy_install tinypie)

$ sudo pip install tinypie
$ tinypie factorial.tp

or clone Git repo and run buildout:

$ git clone git://github.com/rspivak/tinypie.git
$ cd tinypie
$ python bootstrap.py 
$ bin/buildout 
$ bin/tinypie factorial.tp

Main interpreter features:

• Implemented in Python
• Regexp-based lexer
• LL(k) recursive-descent parser
• Parser constructs homogeneous Abstract Syntax Tree (AST)
• Static / lexical scope support.
• Interpreter builds complete scope tree during AST construction.
• Interpeter manages global memory space and function space stack
• Implements external AST visitor
• Forward references support

High-level overview:

Advantages of tree-based interperter over syntax-directed one:

1. Symbol definition and symbol resolution happen at different stages:
   • symbol definition is performed during parsing
   • symbol resolution is performed during interpretation

   This allows forward references and simplifies implementation – parser deals with scope tree only and interpreter deals with memory spaces.

2. More flexible because of AST as an intermediate representation.

Language grammar

Short language reference

• statements: print, return, if, while, and function call
• literals: integers and strings
• operators: <, ==, +, -, *
• Expressions may contain identifiers.

Code examples

1. Factorial

    print factorial(6)               # forward reference

    def factorial(x):                # function definition
        if x < 2 return 1            # if statement
        return x * factorial(x - 1)  # return statement
    .                                # DOT marks the block end

2. WHILE loop

    index = 0

    while index < 10:
        print index
        index = index + 1
    .                                # DOT marks the block end

3. IF ELSE

    x = 10
    if x == 10 print 'Yes'
    else print 'No'

4. Variable lookup in different scopes

    x = 1        # global variable
    def foo(x):  # 'foo' is defined in global memory space
        print x  # prints 3 (parameter value)
    .            # DOT marks the block end
    def bar():   # 'bar' is defined in global memory space
        x = 7    # modify global variable
    .            # DOT marks the block end

    foo(3)       # prints 3
    bar()
    print x      # prints 7 ('bar' modified global variable)

To warm-up after holidays I implemented a simple Syntax-Directed Interpreter for an SQL subset called NO-SO-SQL
Syntax-Directed Interpreter is similar to Syntax-directed translation, but instead of translating source code to another code the interpreter directly executes the source code – no translations, no intermediate representations. It’s suitable for simple DSLs with declarative statements.
The interpreter itself is based on a Syntax-Directed Interpreter from Language Implementation Patterns Ch.9

The main differences are:

1. Lexer is handcrafted
2. LL(k) recursive-descent parser is handcrafted
3. It’s implemented in Python
4. Added # symbol to comment out lines

High-level overview of the interpreter:

Sample file test.nososql:

create table test (primary key name, passwd, quota);
insert into test set name='John', passwd='xxx', quota=100;
insert into test set name='Jim', passwd='yyy', quota=200;
insert into test set name='Test', passwd='test', quota=30;
result = select passwd, quota from test where name='Jim';
print result;

$ bin/nososql test.nososql
yyy 200


Here is a small set of useful commands that I use when dealing with unix timestamps.

To get a current timestamp:

$ date +%s
1292901589

To convert a unix timestamp to human-readable form:

$ date -d @1292901589
Mon Dec 20 22:19:49 EST 2010

To convert a unix timestamp on BSD systems you have to use -r flag:

$ date -r 1292901589

One late evening my colleague and I were sitting in front of my laptop working on some Python code in Emacs. We needed to quickly make some replacements all over the file. The original code looked something like this and we wanted to remove json.dumps call around dictionaries:

def foo():
    return json.dumps({
        'key': 6,
        'q': 7,
        })

def bar():
    return json.dumps({
        'key': 8,
        'q': 9,
        })

We wanted to make it look like this:

def foo():
    return {
        'key': 6,
        'q': 7,
        }

def bar():
    return {
        'key': 8,
        'q': 9,
        }

As you can see a replace regular expression has to deal with code that spans multiple lines and writing Emacs multiline regexp might not be straightforward if you hadn’t written one before.
I’ll tell you right away that we couldn’t figure out quickly how to write that regular expression and had to use “brute force”, but next day I found a solution that works and which I can use in the future when I need to deal with multiline regexp.

Here is what I typed in Emacs:
M-x query-replace-regexp
Query replace regexp: json.dumps((.*(?:^J.*?)*))
Query replace regexp json.dumps((.*(?:^J.*?)*)) with: 1

Let’s take a closer look at the regexp:

1. First .* is reponsible for matching  all characters up to a first newline
2. (?:^J.*?)* – the most important part. It’s a shy group that takes care of matching a newline and everything after the newline up to a next newline.  The shy group is faster and doesn’t introduce new group number so we can use 1 to reference an outer group later. The search inside the group is non greedy due to ? character and allows us to properly handle contents inside multiple json.dumps expressions in the same file. Also note that ^J is not a real character that you need to type – in Emacs to match a newline character when carrying out interactive search/replace you have to enter C-q C-j which will be displayed as ^J.

Here is an EmacsWiki page that helped me figure out some details: MultilineRegexp

Another approach would be to call shell python command on a region. For the regexp I use (?s) to set a flag re.S (dot matches all) to match all characters including newline. Select a region in Emacs buffer or press C-x h to mark the whole buffer and type:

C-u M-|
Shell command on region: python -c “import sys, re; print re.sub(r’(?s)json.dumps((.*?))’, ‘g<1>’, sys.stdin.read())”

I know for many people this is obvious, but I’ve recently seen different developers doing something like that to create an empty file:

$ echo "" > __init__.py

There are “faster” ways to do it:

[alienoid@capricorn ~]$ touch __init__.py
[alienoid@capricorn ~]$ ls -l __init__.py 
-rw-rw-r--. 1 alienoid alienoid 0 Dec 16 07:13 __init__.py
[alienoid@capricorn ~]$ 
[alienoid@capricorn ~]$ rm __init__.py 
[alienoid@capricorn ~]$ 
[alienoid@capricorn ~]$ > __init__.py
[alienoid@capricorn ~]$ ls -l __init__.py 
-rw-rw-r--. 1 alienoid alienoid 0 Dec 16 07:14 __init__.py

I’ve been always fascinated by and interested in interpreters, compilers, and language design. But there is a huge gap between being interested and actually knowing how to implement one whether it’s a simple interpreter, compiler or a programming language.

So couple weeks ago I decided to reduce the gap I had and start with a simple high-level interpreter for a subset of Scheme. Thus after several weekend evenings of code tweaking Serval was born.

Serval is a high-level Scheme interpreter written in Python. It implements a subset of R5RS. The code closely follows Scheme meta-circular evaluator implementation from Ch.4 of the SICP book.

The goals of the project are pretty simple:

1. Self-education (For me there is no better way of gaining and retaining information than by (re)implementing something on my own)
2. To serve as a potential example for other people interested in interpreter implementation, particularly Scheme interpreter.
3. Not a goal per se, but I wanted  Serval to be able to run all examples from “The Little Schemer” book (that was my test target). The project has a test module test_the_little_schemer.py which runs simple meta-circular evaluator from Ch.10 of the book. Yeah, now I can evaluate Scheme code using Scheme evaluator that is interpreted by Scheme interpreter written in Python

To whet your appetite here are some examples of Serval REPL:

1. Defining simple function
   
serval> (define add1 (lambda (x) (+ x 1)))
ok
serval> (add1 9)
10

2. Loading representation from a file
   
serval> (load "/home/alienoid/scheme/the_little_schemer/ch10.ss")
serval> (value '((lambda (x) (cons x x)) 5))
(5 . 5)


To install the interpreter use either pip install

$ sudo pip install serval
$ serval
serval> 

or clone Git repo and run buildout:

$ git clone git://github.com/rspivak/serval.git
$ cd serval/
$ python bootstrap.py 
$ bin/buildout 
$ bin/serval
serval> 

Resources that I found useful when working on Serval:

1. SICP
2. The Little Schemer
3. The Scheme Programming Language
4. Language Implementation Patterns
5. Scheme from Scratch
6. Bob Scheme

In my previous post I wrote about using Ctrl-R for incremental search backward.

It’s about time to move on to incremental search forward.

Default BASH key binding for incremental history search forward is Ctrl-S - no surprises here (if you’re an Emacs user you should feel at home right away without any customizations).

The only issue is that that key binding is shadowed by so called terminal flow control key binding. And you probably experienced that at least once when you accidentally pressed Ctrl-S and your terminal “hang”. To revive terminal after that you have to press Ctrl-Q.

If you run $ stty -a you’ll see those key bindings in a form like start = ^Q; stop = ^S 

There are several workarounds to have your search back:

1. Disable terminal flow control altogether: $ stty -ixon
   Now your Ctrl-S will work like a charm
2. If you want to continue to use terminal flow control then you can just re-bind its Ctrl-S to, let’s say, Ctrl-X allowing BASH Ctrl-S to work as incremental history search forward:
   $ stty stop ^X
3. Bind readline’s  forward-search-history function to another key sequence, for example, Alt-S:
   - $ bind ‘”es”: forward-search-history’
   - or put this line into ~/.inputrc : “es”: forward-search-history

Personally I use (1) / (2) because being an Emacs user it’s natural for me to use Ctrl-S for incremental search forward and it also plays nicely with Ctrl-R – after I cycled backwards through matched commands with Ctrl-R I can cycle forward trough them by simply pressing Ctrl-S, works perfectly without a hitch.

Workaround (3) for some reason doesn’t work as nicely with Ctrl-R, once I press Alt-S after Ctrl-R it clears up typed characters. I put it here just as an alternative for non-Emacs users

When I tell developers about Ctrl-R in BASH usually I get responses literally ranging from “Are you @#$%^&* kidding me? Who doesn’t know it?” to “Thank you. Thank you. Thank you. I owe you a beer”. Well, I made it up about the beer but I figure that people usually imply it

This BASH keybinding is so important for developer’s productivity that it’s well worth repeating it over and over again.

OK, here we go. If you know roughly the contents of the command, but can’t recall where it is in a BASH history list and don’t want to go through that list by hand, then just press Ctrl-R to do a reverse intelligent search.

As you start typing the search goes back in reverse order to the first command that matches the letters you’ve typed. By typing more successive letters you make the match more and more specific.

Once you’ve found the command you have several options:

1. Run it verbatim – just press Enter
2. Edit it before running – you can use arrow keys or different key bindings to navigate to the point you want to edit
3. Cycle through other commands that match the letters you’ve typed – press Ctrl-R successively
4. Quit the search and back to the command line empty-handed – press Ctrl-G

Update:

Added a fourth option that I forgot to mention. Thanks to commenter Taylor for reminding me about it.

BASH has a very concise way to accomplish that: $ > file

[alienoid@capricorn tmp]$ ls -lh archive.log 
-rw-rw-r--. 1 alienoid alienoid 452M Nov 11 19:17 archive.log
[alienoid@capricorn tmp]$ 
[alienoid@capricorn tmp]$ > archive.log 
[alienoid@capricorn tmp]$ 
[alienoid@capricorn tmp]$ ls -lh archive.log 
-rw-rw-r--. 1 alienoid alienoid 0 Nov 11 19:17 archive.log
[alienoid@capricorn tmp]$ 

It’s a good development practice to insert at the beginning of a file copyright, license, and author information. Being lazy I automated the process of insertion using Emacs’ skeleton mode and auto insert mode.

This is an example skeleton configuration:

When I open a new Python file with C-x C-f the header is inserted automatically:

Skeletons support Lisp expressions, so I embedded automatic year generation as well as retrieval of organization’s name, user’s full name, and user’s email address. (You may need to add ORGANIZATION and EMAIL environment variables to your .bash_profile)
M-x skeleton-name will call the skeleton interactively and insert it at the cursor’s position

Alternatively the header insertion automation can be accomplished with yasnippet package, which I use with different programming modes.

Being able to swap lines in Emacs with C-x C-t comes in handy when I edit Python code and want to prettify some imports:

import sys
import os

I move cursor to a line that I want to swap, in this case import os, and press C-x C-t which exchanges the current line and a previous line:

import os
import sys

A very basic and simple operation. And while the same can be accomplished with cutting and pasting – swapping is faster in this case.

When I need to iterate over the lines of all files in sys.argv[1:], I use a fileinput module. This is a typical use of the module.

From the official documentation:

import fileinput
for line in fileinput.input():
    process(line)

This iterates over the lines of all files listed in sys.argv[1:], defaulting to sys.stdin if the list is empty.

Another interesting feature of this module that I occasionally use is in-place filtering that allows to rewrite input files in place. To enable it I pass keyword argument inplace=1 to fileinput.input(). After that standard output is directed to an input file and printing effectively rewrites the file in place:

# inplace.py
import fileinput

def process(line):
    return line.strip() + ' line'

for line in fileinput.input(inplace=1):
    # standard ouput is directed to an input file
    # that's why we need to print here to rewrite the file
    print process(line)

$ cat input1.txt 
first
second
$ 
$ cat input2.txt
third
fourth
$ 
$ python inplace.py input1.txt input2.txt
$ 
$ cat input1.txt 
first line
second line
$ 
$ cat input2.txt
third line
fourth line

I remember once a colleague of mine asked me how to determine if a Python console program got its input directly from a terminal or from a pipe or I/O redirection.

The answer was an isatty function.

In Python every file object has a method isatty that returns True if the file is connected to a tty(-like) device, else False. There is also an os.isatty function.

Here is a test program:

# ttytest.py
import sys

for line in sys.stdin:
    print 'Response: %s' % line.strip()

print 'Connected to a tty device: %s' % sys.stdin.isatty()

Input from a terminal (you may need to press Ctrl-D twice to terminate the input http://bugs.python.org/issue1633941):

$ python ttytest.py 
hello
world
Response: hello
Response: world
Connected to a tty device: True

Input from a pipe:

$ echo -e 'hellonworld' | python ttytest.py
Response: hello
Response: world
Connected to a tty device: False

Input redirection:

$ echo -e 'hellonworld' > data.txt
$ python ttytest.py < data.txt
Response: hello
Response: world
Connected to a tty device: False

Recently I’ve worked on a small project where I needed to pretty print JSON from the command line for quick verification.
For the task I created a pp alias in my .bashrc that worked perfectly (all the code is on the same line):

alias pp='python -c "import sys, json; print json.dumps(
json.load(sys.stdin), sort_keys=True, indent=4)"'

Just today I was re-reading the official Python documentation on a json package and came across a small gem – json.tool module that is used for validation and pretty-printing.
I modified my alias which is now way shorter and does the same job as my old one:

alias pp='python -mjson.tool'

And here is pretty-printing in action:

In one of my previous posts I wrote how I kept track of whitespaces and a column 80 overflow. And in my config you could see that I used delete-trailing-whitespace with write-file-hooks

;; nuke trailing whitespaces when writing to a file
(add-hook 'write-file-hooks 'delete-trailing-whitespace)

What I wanted to use with write-file-hooks was whitespace-cleanup, but somehow it didn’t work for me with that hook (whitespaces were removed when saving a file, but buffer was marked as containing changes).

Thanks to Valeriy Zamarayev who, in his comment to my post, mentioned save hook and whitespace-cleanup. This is what I use now to remove whitespaces when saving a file:

;; nuke whitespaces when writing to a file
(add-hook 'before-save-hook 'whitespace-cleanup)

In the past when I had to edit a file from the command line in my terminal I did one of the following:

• Launched Emacs with the file as a command line parameter. That was usually a slow and annoying process.
• Switched to a running Emacs instance and used C-x C-f to open the file. If the file path was too long it was taking some time to enter it and was annoying too.
• Resorted to Vi in some simple editing cases because it was faster and less annoying (yes, guilty as charged

After being annoyed couple times I set out to find a solution to my problem and it didn’t take long to find out that in Emacs world existed a perfect one: EmacsClient

My problem was solved.

If you’ve never tried EmacsClient before I urge you to do so – it’ll do you some good.

To use it you need to run Emacs in server mode and use a command line utility emacsclient to open a file for editing in an already running Emacs instance. Opening a file for editing in this way is blazingly fast.

I have this line in .emacs to start Emacs in server mode:

(server-start)

You can also start server manually from a running Emacs instance: M-x server-start

Then from the command line send a file to the running instance with emacsclient:
$ emacsclient -n your_file_name

I simplified emacsclient invocation by creating a function and putting it into my .bashrc:

ec() {
    emacsclient -n $1
}

So now whenever I need to edit a file from the command line I just invoke the ec function with a file as a parameter:

$ ec file.txt

P.S.: You can also set environment variable EDITOR to emacsclient to make it a default editor for different utilities relying on the EDITOR variable.

When working in Emacs python mode I keep track of whitespaces and column 80 overflow using a whitespace-mode that is part of Emacs 22+

Here is a snippet from my .emacs:

;; nuke trailing whitespaces when writing to a file
(add-hook 'write-file-hooks 'delete-trailing-whitespace)

;; display only tails of lines longer than 80 columns, tabs and
;; trailing whitespaces
(setq whitespace-line-column 80
      whitespace-style '(tabs trailing lines-tail))

;; face for long lines' tails
(set-face-attribute 'whitespace-line nil
                    :background "red1"
                    :foreground "yellow"
                    :weight 'bold)

;; face for Tabs
(set-face-attribute 'whitespace-tab nil
                    :background "red1"
                    :foreground "yellow"
                    :weight 'bold)

;; activate minor whitespace mode when in python mode
(add-hook 'python-mode-hook 'whitespace-mode)

Because my Emacs is setup to save/restore desktop I’ve added the following lines to play nicely with the whitespace-mode when restoring a previous session:

;; save whitespace-mode variables
(add-to-list 'desktop-globals-to-save 'whitespace-line-column)
(add-to-list 'desktop-globals-to-save 'whitespace-style)

This is how trailing whitespaces, tabs and long lines tails look in my Emacs buffer:

import sys

def main():
    """This line is longer than 80 cloumns ............................."""
    values = [
        # next line contains leading tab
        'qwerty',
        'test',    
        ]

   
if __name__ == '__main__':
    main() # this line contains trailing whitespaces    

UPDATE November 15th, 2011
In Emacs 24 you have to add a face specifier to the whitespace-style to make trailing whitespaces highlighted:

(setq whitespace-line-column 80
      whitespace-style '(face tabs trailing lines-tail))

Long time ago after reading Steve Yegge’s post on Effective Emacs I rebound C-w to ‘backward-kill-word and since then I’ve had the same key binding for both Emacs and BASH.

Now that Conkeror is my default browser it makes sense to have C-w to work the same way as in Emacs and BASH.

This is a snippet that I’ve added to my  ~/.conkerorrc

define_key(text_keymap, 'C-w', 'cmd_deleteWordBackward');

Now I have a uniform key binding across my tools.

Here is a list of BASH key bindings that I use on a daily basis:

Most of these key bindings work in Emacs too.

By default C-m is not bound to Return in Conkeror and I became quite addictive to that binding in Emacs.
It took me a bit to figure it out, but here is a snippet that should go into .conkerorrc to make C-m to work:

require("global-overlay-keymap.js");
define_key_alias("C-m", "return");

My 2 cents for Hidden features of Python

>>> 'xyz' * 3
'xyzxyzxyz'
>>> 
>>> [1, 2] * 3
[1, 2, 1, 2, 1, 2]
>>> 
>>> (1, 2) * 3
(1, 2, 1, 2, 1, 2)

We get the same result with reflected (swapped) operands

>>> 3 * 'xyz'
'xyzxyzxyz'

It works like this:

>>> s = 'xyz'
>>> num = 3

To evaluate an expression s * num interpreter calls s.__mul__(num)

>>> s * num
'xyzxyzxyz'
>>> s.__mul__(num)
'xyzxyzxyz'

To evaluate an expression num * s interpreter calls num.__mul__(s)

>>> num * s
'xyzxyzxyz'
>>> num.__mul__(s)
NotImplemented

If the call returns NotImplemented then interpreter calls
a reflected operation s.__rmul__(num) if operands have different types

>>> s.__rmul__(num)
'xyzxyzxyz'

See http://docs.python.org/reference/datamodel.html#object.__rmul__

Recently I have worked with a REST Web service and had to URL encode a lot of values from the command line.

If all you need is to URL encode data for a POST request you can use curl with –data-urlencode parameter, but I needed to encode values that went into a URL query string.
So for the task I wrote a short code in Python to do just that (there are examples available in BASH or Perl, but this is a pure Python using only standard library):

python -c "import sys, urllib as ul; print ul.quote_plus(sys.argv[1])"

Assign it to an alias by putting single quotes around the expression and you’re good to go:

$ alias urlencode='python -c "import sys, urllib as ul; print ul.quote_plus(sys.argv[1])"'

Example:

$ urlencode 'q werty=/;'
q+werty%3D%2F%3B

And the decode part:

python -c "import sys, urllib as ul; print ul.unquote_plus(sys.argv[1])"

$ alias urldecode='python -c "import sys, urllib as ul; print ul.unquote_plus(sys.argv[1])"'

$ urldecode 'q+werty%3D%2F%3B'
q werty=/;

It took me a while to finish reading the book but here we go.

I wasn’t disappointed by the book and absolutely sure that it delivers on its promise to show that Grok can be an excellent fit for many kinds of Web development projects.

I think I’ll have to finish my Grok based blog now

Grok is largely based on Zope Tookit and that’s a very powerful set of libraries developed over many years that provides support for forms, full text search, authentication, i18n, security, components, persistence, testing, and many more.

Main distinguishing Grok concepts emphasized by the book:

• component architecture
• object database (ZODB)
• object publishing and traversal
• convention over configuration and do not repeat yourself (DRY) principle

The book introduces a simple “to-do” application to the readers. Gradually the application grows with new features like storing data in object database, forms, full text catalog search, security, component architecture.

There is a separate chapter devoted to integration with relational databases. This is especially important topic for Grok because traditionally Zope and Grok were tightly bound to ZODB.

All in all it’s been a pleasant read and I think it’s a great introduction to Grok web framework.

P.S. I may sound very optimistic regarding Grok and Zope, but I can’t do anything about it – I have a soft spot for Grok, it’s an excellent piece of software

Enjoy it!

I’ve been asked by PACKT publishing to review a new book  Grok 1.0 Web Development.

I gladly accepted their proposal. For those interested – yes, they sent me a free PDF version of the book, but I would buy the book anyway because it is about the project I love and associate myself with even though I’ve not been actively participating in it recently.

Personally I think that the release of this book is a major event for the whole Grok community. Having comprehensive documentation for people just starting with a smashing Web framework Grok is of utter importance.

What’s interesting is that I’ve been thinking about writing a book on Grok myself, but Carlos de la Guardia beat me to the punch and after skimming through the book I’m glad he did

What I liked instantaneously about the book is the presence of a chapter on integration with relational databases, the topic that in the Zope world was kind of obscure in the past.

When I finish reading the book I’ll post a full review so stay tuned and meanwhile you can take a sneak peak at the book by reading a freely available Chapter No 5: Forms

Got my shiny new Nokia N900 – to say that I’m happy is an understatement

For me as a long standing Linux user and Pythonista N900 has an unassailable advantage – its Maemo platform is Linux based and I can program in Python for the platform.

So it’s time to get my feet wet and learn some Maemo programming. I decided to take a stab at writing a small Anki client using Python and Qt. As a result I have a working but still very raw prototype called KTAnki that successfully runs on my N900.

For KTAnki I used PyQt4 library, though I should take a closer look at Nokia’s own bindings – PySide.

Here are some resources that I found helpful when prototyping my first Maemo PyQt application:

1. N900 USB Networking
2. PyMaemo QuickStart Guide
3. Getting started with PyQt for Maemo
4. PyQt Application Development On Maemo
5. Hildonized applications for Maemo 5 with Qt4.5.3 and Qt4.6
6. Rapid GUI Programming with Python and Qt
7. Qt Reference Documentation

I’ve added a new repository for CSSBuilder at http://github.com/rspivak/cssbuilder

CSSBuilder is a command line utility and library written in Python for embedding Data URIs into CSS files.

You can find more information about Data URI at:

http://en.wikipedia.org/wiki/Data_URI_scheme

http://www.nczonline.net/blog/2009/10/27/data-uris-explained/

http://www.phpied.com/data-urls-what-are-they-and-how-to-use/

The other day I had to explain why in Python in contrast to Perl/Lisp/Scheme one can’t rebind variable outside of the local scope besides the global scope, i.e. the following function will result in an UnboundLocalError exception in Python 2.x

def outer():
    count = 0
    def inner():
        count += 1 # rebinding
    inner()
    print count

>>> outer()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "scope.py", line 8, in outer
    inner()
  File "scope.py", line 6, in inner
    count += 1 # rebinding
UnboundLocalError: local variable 'count' referenced before assignment

There are several workarounds.

For one thing it is possible to use a variable holding a mutable object:

def outer():
    ns = dict(count=0)
    def inner():
        ns['count'] += 1 # rebinding
    inner()
    print ns['count']

>>> outer()
1

Another approach is to use a function object itself to store the variable:

def outer():
    outer.count = 0
    def inner():
        outer.count += 1 # rebinding
    inner()
    print outer.count

>>> outer()
1

The good news is that there is no need for workarounds anymore in Python 3 which introduced nonlocal keyword:

def outer():
    count = 0
    def inner():
        nonlocal count
        count += 1 # rebinding
    inner()
    print(count)

>>> outer()
1

Recently I’ve been working with CSS files and wanted to replace color definitions with downcased equivalents, i.e. #FFFFFF; would become #ffffff;

To do that I used pretty well known feature of Emacs, namely use of Lisp expressions in the replacement string of replace-regexp function.

Commands entered in minibuffer:

M-x replace-regexp RET (#.*)  RET ,(downcase 1) RET

The only problem with that code is that if you have variable case-fold-search in Emacs set to true (which is the case in my Emacs by default) replace-regexp will report about successful replace but string won’t be downcased anyway. To quickly switch off case-fold-search I used M-: (setq case-fold-search nil) and after that downcasing worked as expected.

To use IPython shell from Grok based project is easy, just add ipython to eggs directive in your buildout.cfg

My part with ipython added looks like this:

[app]
recipe = zc.recipe.egg
eggs = grok-awesome
       ipython
       Paste
       PasteScript
       PasteDeploy
interpreter = python-console

But I wanted to be able to inspect Grok instance and use path TAB auto-completion to navigate ZODB hierarchy.

Inspired by existing IPython profile fo Zope2 I whipped out basic implementation for Grok.

It’s available under grok-ipython. Now all IPython power can be used when inspecting objects in Grok instance:

(mygrok)[alienoid@capricorn grok-awesome]$ bin/ipython -p grok
IPython shell for Grok.

Bound object names:
-------------------
  root
  ctx

Bound command names:
--------------------
  cdg / ;cdg
  lsg / ;lsg
  pwdg
  sync
  commit

Grok IPython 0.9.1   Py 2.5.2

In [1]: root
Out[1]: <zope.app.folder.folder.Folder object at 0x9003e6c>

In [2]: ctx
Out[2]: <zope.app.folder.folder.Folder object at 0x9003e6c>

In [3]: ;cdg blog
------> cdg("blog")
Out[3]: u'/blog'

In [4]: ctx
Out[4]: <grokawesome.blog.Blog object at 0x81d2d6c>

In [5]: ;cdg 
my-first-post  second-post    

In [5]: ;cdg my-first-post
------> cdg("my-first-post")
Out[5]: u'/blog/my-first-post'

In [6]: ctx
Out[6]: <grokawesome.entry.BlogEntry object at 0xb7d556ac>

In [7]: pwdg
Out[7]: u'/blog/my-first-post'

In [8]: ctx?
Type:           BlogEntry
Base Class:     <class 'grokawesome.entry.BlogEntry'>
String Form:    <grokawesome.entry.BlogEntry object at 0xb7d556ac>
Namespace:      Interactive
Length:         0
File:           /home/alienoid/mygrok/grok-awesome/src/grokawesome/entry.py
Docstring:
    <no docstring>


In [9]: ctx??
Type:             BlogEntry
Base Class:       <class 'grokawesome.entry.BlogEntry'>
String Form:   <grokawesome.entry.BlogEntry object at 0xb7d556ac>
Namespace:        Interactive
Length:           0
File:             /home/alienoid/mygrok/grok-awesome/src/grokawesome/entry.py
Source:
class BlogEntry(grok.Container):
    grok.implements(IBlogEntry)

    portal_type = 'blogentry'
    created = property.DCProperty('created')
    modified = property.DCProperty('modified')

    def __init__(self, title, content,
                 summary=u'',
                 categories=None,
                 allow_comments=False):
        super(BlogEntry, self).__init__()
        self.title = title
        self.content = content
        self.summary = '' if summary is None else summary
        self.categories = [] if categories is None else categories
        self.allow_comments = allow_comments

I have added TinyMCE widget to grok-awesome for editing html and it was easy as pie.

If you want TinyMCE in your Grok based project it can be achieved with following steps:

1. Add dependencies to setup.py into install_requires section:

                        'hurry.tinymce',
                        'hurry.zopetinymce',

hurry.tinymce allows you to include TinyMCE into your project without additional burden, you don’t need to do anything. Well, almost – just make sure to call:
from hurry.tinymce import tinymce
tinymce.need()
to trigger inclusion of TinyMCE in the web page.

2. Create custom widget which calls tinymce.need() as noted above and initializes TinyMCE

from zope.app.form.browser import TextAreaWidget
from hurry.tinymce import tinymce

template = """%(widget_html)s
<script type="text/javascript">
  tinyMCE.init({
    mode : "exact",
    elements: "%(elements)s",
    theme: "advanced"
  });
</script>"""

class TinyMCEWidget(TextAreaWidget):
    """Widget to edit html using TinyMCE WYSIWYG editor."""

    def __call__(self, *args, **kw):
        widget_html = super(TinyMCEWidget, self).__call__(*args, **kw)
        tinymce.need()
        return template % {'widget_html': widget_html,
                           'elements': self.name,
                           }

3. In your add/edit form use your custom widget for the field you want and off you go:

form_fields['content'].custom_widget = TinyMCEWidget

The other day I had to decode strings containing hex numbers into plain integers.

I could use built-in int([ x [, radix]]) with radix=16 to convert a string to an integer and forget about it, but I needed to decode pairs from the hex number so that in the end from the string ‘FFEEDD’ I get 255, 238, 221

But wouldn’t you know it, there is a standard solution to get pairs in Python and it’s built-in zip function.

It’s a well known function for experienced developers, but there is one documented feature of it that doesn’t always catch one’s eye and it’s a general form for clustering a data series into n-length groups zip(*[iter(s)]*n) which is described in official docs for zip built-in.

With that in mind solution to my problem boils down to a one-line decoder:

>>> s = 'FFDDEE'
>>> [int(''.join(pair), 16) for pair in zip(*[iter(s)]*2)]
[255, 221, 238]

For those wondering how it works:

1) iter(s) creates iterable from the s

>>> iter(s)
<iterator object at 0x8a3c2cc>

2) [iter(s)] * 2 creates list of 2 iterators that all use the same underlying iterable iter(s). The important thing is that created iterators share the same iterable, so when some data is consumed by first iterator that data becomes unavailable for the second iterator and vice versa.

You can see from the output that iterator objects are located at the same address. If you are new to this behavior you can read on about shallow copies.

>>> [iter(s)]*2
[<iterator object at 0x8a3c2cc>, <iterator object at 0x8a3c2cc>]

3) leading * in zip’s parameter unpacks list from step (2) into two separate parameters for zip function. Basically it becomes zip(iterator, iterator).

More about unpacking argument list you can read in official docs.

>>> zip(*[iter(s)]*2)
[('F', 'F'), ('D', 'D'), ('E', 'E')]

I often need to refresh page in Firefox when I edit page template or javascript file. This is especially true when working on grok-awesome.

Though I use tiling window manager StumpWM and it’s very easy to switch to running Firefox with keyboard shortcut and make refresh, it’s even easier not to leave focus from Emacs window and just send command to Firefox to refresh itself.
For that I use:

• MozRepl firefox plugin to allow connect to Firefox’s REPL. After installation in Tools -> MozRepl menu I chose ‘Activate on startup’ for convenience
• MozRepl Emacs integration (If you use nXhtml package in Emacs you already have that integration and can skip this part)

Command to refresh Firefox is BrowserReload();

Final touch is small anonymous function to send that command to mozrepl, keybinding to C-x p and we’re all set:

(global-set-key (kbd "C-x p")
                (lambda ()
                  (interactive)
                  (comint-send-string (inferior-moz-process)
                                      "BrowserReload();")))

Now anytime just press C-x p and refresh command will be sent to Firefox starting along the way mozrepl session in your Emacs if it’s not been already started.

I’ve started developing blog software for “A Smashing Web Framework” Grok.

The reasons behind that:

• There is no full-featured blog software for Grok. There is a simple version Grokstar, but as for me it doesn’t fit the bill.
• I created simple blog for Zope3 several years ago and let it languish. Now it’s time to revive it taking into account how Grok simplifies development with Zope3 technologies.
• I’m not completely happy with WordPress. For one thing, I’d like to have easy-to-use REST API so I can use it in my Emacs to make posts.
• I simply love Grok. You should try it and you’ll love it too.

My fledgling project lives on GitHub and it’s called grok-awesome

Ever wondered how to make IPython colors be nice on dark background of your dark (no pun intended) Emacs theme like blackboard ?

Just add to your .emacs

(setq py-python-command-args '("-pylab" "-colors" "Linux"))

before you load ipython package

(setq py-python-command-args '("-pylab" "-colors" "Linux"))
(require 'ipython)

Now IPython uses ‘Linux’ color scheme which is suitable for dark background with light fonts instead of default ‘LightBG’.

Here it is:

Python 2.5.2 (r252:60911, Sep 30 2008, 15:41:38)
Type "copyright", "credits" or "license" for more information.

IPython 0.9.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

  Welcome to pylab, a matplotlib-based Python environment.
  For more information, type 'help(pylab)'.

In [1]: d = dict()

In [2]: ?d
Type:           dict
Base Class:     <type 'dict'>
String Form:    {}
Namespace:      Interactive
Length:         0
Docstring:
    dict() -> new empty dictionary.
    dict(mapping) -> new dictionary initialized from a mapping object's
        (key, value) pairs.
    dict(seq) -> new dictionary initialized as if via:
        d = {}
        for k, v in seq:
            d[k] = v
    dict(**kwargs) -> new dictionary initialized with the name=value pairs
        in the keyword argument list.  For example:  dict(one=1, two=2)

In [3]: 

I’ve just finished reading excellent book Release It!: Design and Deploy Production-Ready Software

If you are involved or plan to be involved in developing any system that must be available 24 x 7 x 365 then you owe it to yourself to read this book.

The key ideas you should understand are:

• system will fail

• life of software only begins when you release it into production

With that in mind the book is a treasure trove of useful advices how to make your system scale and succeed in production.

The book resonates with my own personal experience in developing and maintaining critical Python application and many things that Michael Nygard describes I learned the hard way: blocked threads, socket timeouts, integration point failures and latency, asynchronous handling, caching, database growth and of course “cold shivers” you get when the system goes down.

While book is sprinkled with notes about Java related issues it’s nevertheless technology agnostic which makes it a valuable source of information for every developer.

It gets two thumbs up from me.

Sometimes you may need to have nested dictionaries initialized without much hassle.

“Autovivication” comes in handy. What you want to achieve is:

>>> d = dict()
>>> d['q']['w']['r'] = 777

Of course what you’ll get will be:

>>> d['q']['w']['r'] = 777
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'q'

If you don’t need deeply nested dictionaries defaultdict with dict default factory will do the job:

>>> from collections import defaultdict
>>> d = defaultdict(dict)
>>> d
defaultdict(<type 'dict'>, {})
>>> d['a']['b'] = 33
>>> d
defaultdict(<type 'dict'>, {'a': {'b': 33}})
>>> d['a']['b']
33

Here are another ways to have unlimited level of nested dictionaries.
1) Define custom class inheriting from built-in dict:

class default_dict(dict):
    def __missing__(self, key):
        self[key] = value = self.__class__()
        return value

>>> d = default_dict()
>>> d
{}
>>> d['q']['w']['r'] = 777
>>> d
{'q': {'w': {'r': 777}}}
>>> d['q']['w']['r']
777

2) Recursive function + defaultdict:

def make_dict():
    return defaultdict(make_dict)

>>> from collections import defaultdict
>>> d = defaultdict(make_dict)
>>> d
defaultdict(<function make_dict at 0xb7c5295c>, {})
>>> d['q']['w']['r'] = 777
>>> d
defaultdict(<function make_dict at 0xb7c5295c>, {'q': defaultdict(<function make_dict at 0xb7c5295c>, {'w': defaultdict(<function make_dict at 0xb7c5295c>, {'r': 777})})})
>>> d['q']['w']['r']
777

3) And for fun you can use Y Combinator (in short – making anonymous lambda recursive) which is actually as (2) but you don’t have to have recursive function defined:

def Y(g):
    return ((lambda f: f(f))
            (lambda f:
                 g(lambda x: f(f)
                   (x))))

defdict = Y(lambda mk_dict:
                lambda x=None: defaultdict(lambda x=None: mk_dict(x)))

>>> d = defdict()
>>> d
defaultdict(<function <lambda> at 0xb7c52a74>, {})
>>> d['q']['w']['r'] = 777
>>> d
defaultdict(<function <lambda> at 0xb7c52a74>, {'q': defaultdict(<function <lambda> at 0xb7c56c6c>, {'w': defaultdict(<function <lambda> at 0xb7c56a74>, {'r': 777})})})
>>> d['q']['w']['r']
777

Recent Erlang releases (since R12B) have array module included.

Now binary search algorithm can be implemented quite efficiently with functional arrays:

-module(bsa).
-export([binsearch/2]).

binsearch(Arr, Key) ->
    binsearch(Arr, Key, 0, array:size(Arr)).

binsearch(Arr, Key, LowerBound, UpperBound) ->
    Mid = (LowerBound + UpperBound) div 2,
    Item = array:get(Mid, Arr),
    if
        UpperBound < LowerBound -> -1;
        Key < Item ->
            binsearch(Arr, Key, LowerBound, Mid-1);
        Key > Item ->
            binsearch(Arr, Key, Mid+1, UpperBound);
        true ->
            Mid
    end.

Running time for lists based binary search (just for comparison):

56> {Time, Val} = timer:tc(search, binsearch,
56>                        [lists:seq(1, 1000000), 1000000]).
{506513,  1000000}

Running time for array based binary search:

57> f().
ok
58> {Time, Val} = timer:tc(bsa, binsearch,
58>               [array:from_list(lists:seq(1, 1000000)), 1000000]).
{17,  999999}

Previous parts: Intro, Part I, Part II, Part III and Part IV

List comprehensions.

List comprehensions is very familiar topic for Python programmer. It’s a syntactic sugar which provides a succinct notation for producing elements in list.

Erlang:

[Expr || Qualifier1,...,QualifierN]

Expr – is an arbitrary expression
Qualifier is either a generator or a filter.
A generator is of form Pattern <- ListExpr where ListExpr evaluates to a list of terms.
A filter expression should evaluate to true or false.

1> [X || X <- [1,2,3,4,5,6], X > 3].
[4,5,6]

This is read as: list of X such that X is taken from list [1,2,3,4,5,6] and X is greater than 3.

In foregoing example: X <- [1,2,3,4,5,6] is a generator and X > 3 is a filter.

Several filters can be combined:

2> [X || X <- [1,a,2,3,b,4,5,6], X > 3].
[a,b,4,5,6]
3> [X || X <- [1,a,2,3,b,4,5,6], integer(X), X > 3].
[4,5,6]

Generator part may act as a filter in list comprehension:

4> [X || {X, a} <- [{1, a}, {2, b}, {3, a}, erlang]].
[1,3]

Generators can be combined too in list comprehensions. This is the Cartesian product of two lists:

5> [{X, Y} || X <- [1, 2, 3], Y <- [a, b]].
[{1,a},{1,b},{2,a},{2,b},{3,a},{3,b}]

This is how elegantly quicksort algorithm is solved with the help of list comprehensions in Erlang:

-module(sort).
-export([qsort/1]).

qsort([]) -> [];
qsort([Pivot|Tail]) ->
    qsort([X || X <- Tail, X < Pivot])
        ++ [Pivot] ++
        qsort([X || X <- Tail, X >= Pivot]).

Compile and run:

6> c("/home/alienoid/dev/erlang/sort", [{outdir, "/home/alienoid/dev/erlang/"}]).
{ok,sort}
7> sort:qsort([7, 5, 9, 3, 6]).
[3,5,6,7,9]

Usage of list comprehensions instead of some higher-order functions:

8> lists:map(fun(X) -> X*2 end, [1, 2, 3]).
[2,4,6]
9> [X*2 || X <- [1, 2, 3]].
[2,4,6]

10> lists:filter(fun(X) -> X rem 2 == 0 end, [1, 2, 3, 4]).
[2,4]
11> [X || X <- [1, 2, 3, 4], X rem 2 == 0].
[2,4]

Python:

[expression for expr1 in seq1 if cond1
            for expr2 in seq2 if cond2
            for exprN in seqN if condN]

this actually transforms to:

for expr1 in seq1:
    if not cond1:
        continue
    for expr2 in seq2:
        if not cond2:
            continue
        for exprN in seqN:
            if not condN:
                continue

So in Python Cartesian product of two lists [1, 2, 3 ] and ['a', 'b'] will look like:

>>> [(x, y) for x in [1, 2, 3] for y in ['a', 'b']]
[(1, 'a'), (1, 'b'), (2, 'a'), (2, 'b'), (3, 'a'), (3, 'b')]

Variable bindings and scope rules in List Comprehensions:

Current Python 2.x “leaks” loop variable into surrounding scope. This should be solved in Python 3.0

>>> x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined
>>> vlist = [x for x in 'abc']
>>> vlist
['a', 'b', 'c']
>>> x
'c'

To avoid leaking loop variable into surrounding scope we can use generator expression which doesn’t “leak”:

>>> vlist = list(x for x in 'abc')
>>> vlist
['a', 'b', 'c']
>>> x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined

Erlang has following scope rules for variables in list comprehensions(from Erlang manual):

• all variables which occur in a generator pattern are assumed to be “fresh” variables
• any variables which are defined before the list comprehension and which are used in filters have the values they had before the list comprehension
• no variables may be exported from a list comprehension

So, Erlang doesn’t “leak” variables:

1> [X || X <- [1, 2, 3]].
[1,2,3]
2> X.
** 1: variable 'X' is unbound **

Be careful, sometimes you need to move some pattern matching operations into filter:

-module(listcomp).
-export([select/2]).

%% This select produces following warning during compilation:
%% Warning: variable 'X' shadowed in generate
select(X, List) ->
    [Y || {X, Y} <- List].

The function doesn’t produce desired result:

1> listcomp:select(b, [{b, 1}, {c, 2}, {b, 3}]).
[1,2,3]

So we modify generator and move pattern matching operation into filter to achieve what we need:

-module(listcomp).
-export([select/2]).

%% Moving X from pattern matching into filter
select(X, List) ->
    [Y || {X1, Y} <- List, X == X1].

2> listcomp:select(b, [{b, 1}, {c, 2}, {b, 3}]).
[1,3]

As you saw, while different in syntax, both languages provide valuable language construct which allows to write shorter and more elegant programs.

Sometimes after I copy-paste code snippets from web page into emacs buffer I get text with superfluous blank lines delimiting code.

Marking region and running M-x flush-lines RET ^$ RET quickly makes it better by removing those pesky blank lines. This is, of course, makes sense only if pasted code doesn’t contain blank lines you want to preserve.

Update:
You may need to use something more complex than simple ^$ (something like ^W*$ as regexp) if “blank” line contains some whitespace characters.

Example before:

-export([test1/0,

         test2/0,

         test3/0,

         double/1]).

After marking region and applying M-x flush-lines RET ^W*$ RET (in your case just ^$ may be sufficient):

-export([test1/0,
         test2/0,
         test3/0,
         double/1]).

Well-known incremental search forward(isearch-forward Lisp function) which is invoked with C-s has also additional useful key-bindings which I use often to instruct what to search. After you pressed C-s in buffer they are:

1. C-w to yank next word or character in buffer onto the end of the search string, and search for it. You can repeatedly press C-w consuming next word or character.
2. C-y to yank rest of line onto end of search string and search for it.
3. M-n/M-p to search for the next/previous item in the search ring.

As usual for more information read help (C-h k C-s or C-h f isearch-forward RET ).

This interactive Emacs Lisp function was very handy for me during some log files analysis.

You mark region, invoke function, provide regexp and voila – your region is aligned.

Example data from align-regexp help:

Fred (123) 456-7890
Alice (123) 456-7890
Mary-Anne (123) 456-7890
Joe (123) 456-7890

Let’s say we want to align region on “(” character. For that we mark region we want to align and invoke function with: M-x align-regexp RET ( RET
Result of such invocation will be:

Fred      (123) 456-7890
Alice     (123) 456-7890
Mary-Anne (123) 456-7890
Joe       (123) 456-7890

Previous parts: Intro, Part I, Part II and Part III

Funs – sequel

Continuing previous post about functions let’s remember about Fun. I already mentioned it in Part I and promised to show its advanced forms. So, here we go.

Fun with one clause:

1> Double = fun(X) -> X * 2 end.
#Fun<erl_eval.6.56006484>
2> Double(2).
4

But Fun actually has the same function declaration syntax as regular function, except that it has no name in declaration, so we can have this:

3> Big = fun (X) when X > 100 -> true;
3>           (X) -> false
3>       end.
#Fun<erl_eval.6.56006484>
4> lists:filter(Big, [20, 30, 120, 130]).
[120,130]

As you see our Fun has two clauses separated by semicolon(;) and first clause has also guard, ie syntax is the same as in regular function declaration.

There are also following fun expressions that can be used:

1. fun FunctionName/Arity (FunctionName should point to local function defined in the same module with our fun)
2. fun Module:FunctionName/Arity (FunctionName should be exported from module Module)

fun FunctionName/Arity is just syntactic sugar for fun (X1, ..Xn) -> FunctionName(X1, .., Xn) end.

Let’s take a look on foregoing in action and create mymath.erl

-module(mymath).
-export([test1/0,
         test2/0,
         test3/0,
         double/1]).

test1() -> lists:map(fun(X) -> X * 2 end, [1, 2, 3]).

test2() -> lists:map(fun double_local/1, [1, 2, 3]).

test3() -> lists:map(fun mymath:double/1, [1, 2, 3]).

%% helper function, not exported
double_local(X) ->
    X * 2.

%% helper function, exported
double(X) ->
    X * 2.

Compile and test it (I’m compiling as usual in Emacs with C-c C-k):

1> c("/home/alienoid/dev/erlang/mymath", [{outdir, "/home/alienoid/dev/erlang/"}]).
{ok,mymath}
2> mymath:test1().
[2,4,6]
3> mymath:test2().
[2,4,6]
4> mymath:test3().
[2,4,6]

Also tuple of type {Module, FunctionName} is interpreted as a fun Module:FunctionName/Arity which you already saw. This usage is deprecated, but to be complete here is an example:

5> Append = {lists, append}.
{lists,append}
6> Append([1, 2], [3, 4, 5]).
[1,2,3,4,5]

As you already noted Fun in Erlang is “richer” than lambda in Python.

Built-in functions, BIFs

Both Erlang and Python provide built-in functions that are always available and do not require explicit usage of module name to use them.

Python:

Read more about builtins in library reference.

In Python shell you can inspect in addition contents of __builtins__ module if you need:

>>> import pprint
>>> pprint.pprint(dir(__builtins__))
['ArithmeticError',
 'AssertionError',
 ...
 'type',
 'unichr',
 'unicode',
 'vars',
 'xrange',
 'zip']

Erlang:
Most of the built-in functions belong to module erlang and they are auto-imported.
To get list of all BIFs with description read man erlang.
In Eshell you can get list of functions that belong to erlang module with m(Module). command:

1> m(erlang).
Module erlang compiled: No compile time info available
Compiler options:  []
Object file: preloaded
Exports:
'!'/2                         list_to_existing_atom/1
'$erase'/1                    list_to_float/1
'$erase'/0                    list_to_integer/2
'$get'/1                      list_to_integer/1
'$get'/0                      list_to_pid/1
'$get_keys'/1                 list_to_tuple/1
'$put'/2                      load_module/2
'*'/2                         loaded/0
'+'/1                         localtime/0
'+'/2                         localtime_to_universaltime/1
'++'/2                        localtime_to_universaltime/2
...
ok

Take into account though that not all functions in above list that you’ll see are auto-imported and may require usage of module: prefix. Again, for more information read man erlang (either in command line with erl -man erlang or if you use Emacs take a look at Erlang man pages in Emacs)

Macros

Python does not have them, Erlang does. In Erlang they are not as powerful as in Common Lisp, but more like in C.

They are defined in following form:

-define(Const, Replacement).
-define(Func(Var1,...,VarN), Replacement).

Whenever erlang preprocessor encounters expression of form ?MacroName it expands corresponding macro.
Define utils.erl:

-module(utils).
-export([foo/0]).
-define(TIMEOUT, 100).
-define(FUNCMACRO(X, Y), {X, x, Y, y}).

foo() ->
    io:format("TIMEOUT = ~p~n", [?TIMEOUT]),
    ?FUNCMACRO(3, 7).

Compile and run:

1> utils:foo().
TIMEOUT = 100
{3,x,7,y}

There are also some predefined macros:

?MODULE

The name of the current module.

?MODULE_STRING.

The name of the current module as a string.

?FILE.

The file name of the current module.

?LINE.

The current line number.

?MACHINE.

The machine name, 'BEAM'.

Having ?MODULE we can rewrite earlier example with Fun as:

-module(mymath).
-export([test1/0,
         test2/0,
         test3/0,
         double/1]).

test1() -> lists:map(fun(X) -> X * 2 end, [1, 2, 3]).

test2() -> lists:map(fun double_local/1, [1, 2, 3]).

%% test3() -> lists:map(fun mymath:double/1, [1, 2, 3]).
test3() -> lists:map(fun ?MODULE:double/1, [1, 2, 3]).

%% helper function, not exported
double_local(X) ->
    X * 2.

%% helper function, exported
double(X) ->
    X * 2.

And as usual compile and run to see that result is the same:

1> c("/home/alienoid/dev/erlang/mymath", [{outdir, "/home/alienoid/dev/erlang/"}]).
{ok,mymath}
2>  mymath:test1().
[2,4,6]
3>  mymath:test2().
[2,4,6]
4>  mymath:test3().
[2,4,6]

In addition there is also flow control in macros. Some of corresponding macro directives are:

• ifdef(Macro). – Evaluate the following lines only if Macro is defined.
• else. – It’s used after ifdef or ifndef
• endif. - Marks the end of ifdef or ifndef directive

Example in utils.erl:

-module(utils).
-export([foo/0]).

-ifdef(debug).
-define(LOG(Msg), io:format("{~p:~p}: ~p~n", [?MODULE, ?LINE, Msg])).
-else.
-define(LOG(Msg), true).
-endif.

foo() ->
    ?LOG("Debug is enabled").

To turn LOG macro on debug should be defined. This can be achieved from command line with erlc -Ddebug utils.erl or from Eshell using c/2 function:

1> c(utils, {d, debug}).
{ok,utils}
2> utils:foo().
{utils:11}: "Debug is enabled"
ok

To turn LOG macro off debug should be omitted:

3> c(utils).
{ok,utils}
4> utils:foo().
true

To be continued.

Sometimes I need to create tables in Emacs as it was in post describing comparison and arithmetic operations in Python and Erlang.

Over time I became aware of several means to create tables and export them in different formats in Emacs(at the moment I’m personally interested in plain ASCII and HTML):

1. Excellent org-mode has built-in table editor which allows easily format tables in plain ASCII
2. table.el with its advanced ability to construct table layout

Both modes (technically though Table mode in table.el is not mode) are part of modern Emacs and both allow export table in HTML format.

To quickly create tables personally I prefer table.el which provides also following features:

1. A cell can be split horizontally and vertically.
2. A cell can span into an adjacent cell.

Use C-h b in table to get self-descriptive information about table.el key bindings. In contrast to org-mode(or its orgtbl-mode minor mode) table.el has no special bindings to copy row/cell and yank it. But this is easily achieved by using usual region commands for rows and rectangular commands for cells.

Previous parts: Intro, Part I and Part II

Today we are going to take a look at functions.
In Erlang there is no direct correspondence to what we have in Python:

1. default values, keyword arguments and formal parameter of form **name

def foo(key, name='ruslan', type='unknown', **kw):
    print 'key = %s, name = %s, type = %s' % (key, name, type)
    for key, val in kw.items():
        print '%s = %s' % (key, val)

>>> foo(1)
key = 1, name = ruslan, type = unknown
>>> foo(1, type='known', x=1, y=10)
key = 1, name = ruslan, type = known
y = 10
x = 1

2. formal parameter of form *name

def double_sum(*args):
    return sum(x * 2 for x in args)

>>> double_sum(2, 4, 5)
22

Let’s take a look at Erlang’s function declaration syntax (a bit modified from Erlang Reference Manual):

1. Function declaration consists of function clauses separated by semicolons (;) and terminated by period (.)
2. Function clause consists of head and body separated by ->
3. A clause head consists of function name (which is atom), argument list(each argument is pattern) and optional guard sequence beginning with keyword when.
4. A clause body consists of sequence of expressions separated by comma (,)

Name(Pattern11,…,Pattern1N) [when GuardSeq1] ->
Body1;
…;
Name(PatternK1,…,PatternKN) [when GuardSeqK] ->
BodyK.

where Body is in form:
Expr1,
Expr2,
…
ExprN

Number of arguments in function is called arity. Function is uniquely identified by combination of module name, function name and function’s arity and this form is often written as m:f/N. Two functions in the module with the same name, but different arity
are completely different functions, remember it.

Enough dry theory, let’s define our Erlang factorial function in module mymath.erl :

-module(mymath).
-export([factorial/1]).

factorial(0) ->
    1;
factorial(N) ->
    N * factorial(N-1).

What we have here:

1) function declaration of one function called factorial
2) there are two clauses
- first clause separated by semicolon (;)

factorial(0) ->
    1;

- second and final clause terminated with period (.)

factorial(N) ->
    N * factorial(N-1).

3) Every clause (first and second) has correspondingly head and body. Head consists of function’s name, in our case it’s factorial and after follows arguments list which contains patterns. In first clause pattern is 0, in second – N.

How this works:

When function is entered, the function clauses are pattern matched against passed arguments. Pattern matching happens in order functions are defined in module. So when you enter

1> mymath:factorial(5).
120

first 5 is matched against clause factorial(0) and fails as 5 != 0, then 5 is matched against N in second clause and pattern matching succeeds. As a result unbound variable N in pattern becomes bound and gets value 5. This leads to execution of body of second clause, namely N * factorial(N-1). This happens recursively until N becomes 0 and first clause factorial(0) is executed terminating recursion.
The same factorial function will look in Python without pattern matching like:

def factorial(n):
    if n == 0:
        return 1
    return n * factorial(n-1)

In Python’s version we “hardcoded” branching in function’s body.

The benefits of pattern matching are that when conditions inside function become complex you may do your job better and write less code using pattern matching and also if you have big function this may mean that you do not have to modify body of your function but to add another clause with corresponding pattern.

Earlier I mentioned that function clauses are pattern matched in order they are defined in module. For our function this is important. If we change order of clauses this will lead to problems, as the condition to terminate recursion never appear:

-module(mymath).
-export([factorial/1]).

factorial(N) ->
    N * factorial(N-1);
factorial(0) ->
    1.

When compiling we will get warning (use C-c C-k in Emacs or c(mymath). in Eshell to compile)

2> c("/home/alienoid/dev/erlang/mymath", [{outdir, "/home/alienoid/dev/erlang/"}]).
/home/alienoid/dev/erlang/mymath.erl:6: Warning: this clause cannot match because a previous clause at line 4 always matches
{ok,mymath}

indicating that second clause will never match, but still you can call the function (it will most likely to crash after eating all your available memory).

You can easily avoid that problem introducing guard that starts with keyword when in function clause’s head:

-module(mymath).
-export([factorial/1]).

factorial(N) when N > 0 ->
    N * factorial(N-1);
factorial(0) ->
    1.

Now you can put those two clauses in any order you like. I should also add that if during pattern matching no matched function clause is found a function_clause run time error will occur:

3> mymath:factorial(-4).

=ERROR REPORT==== 24-Sep-2007::01:53:25 ===
Error in process <0.29.0> with exit value: {function_clause,[{mymath,factorial,[-4]},{erl_eval,do_apply,5},{shell,exprs,6},{shell,eval_loop,3}]}

** exited: {function_clause,[{mymath,factorial,[-4]},
                             {erl_eval,do_apply,5},
                             {shell,exprs,6},
                             {shell,eval_loop,3}]} **

What is guard anyway ? Guards make pattern matching more powerful, they can be used to perform different tests and comparison operations on variables in a pattern as you could see. You can have several guard expressions in clause separated by commans (,). The guard containing several guard expressions evaluates to true only if every guard expression evaluates to true. Example of guard containing several guard expressions:

-module(mymath).
-export([factorial/1]).

factorial(N) when is_integer(N), N > 0 ->
    N * factorial(N-1);
factorial(0) ->
    1.

For above definition calling mymath:factorial(5) reads as: choose clause factorial(N) when (N is integer AND N > 0)

There are also guard sequences which have form guard1; guard2; …; guardN and guard sequence evaluates to true if at least one guard is evaluated to true:

-module(mymath).
-export([factorial/1]).

factorial(N) when is_integer(N), N > 0; is_list(N) ->
    N * factorial(N-1);
factorial(0) ->
    1.

For above definition calling mymath:factorial(5) reads as: choose clause factorial(N) when (N is integer AND N > 0) OR N is list.

You have probably noted that the way we defined our factorial function means that it will be calling itself recursively and this will use stack and stack is not infinite resource. We need to do something about our function and this means to make it tail recursive. When last expression in function body is function call this means that call is tail recursive and this is how infinite loop may be done without consuming stack resources. So here is our new tail-recursive function factorial that uses accumulator to pass value of previous calculation:

-module(mymath).
-export([factorial/1]).

%% this one is exported, arity is 1
factorial(N) ->
    factorial(N, 1).

%% helper function, not exported, arity is 2
%% tail-recursive, uses accumulator to store
%% calculation between invocations
factorial(N, Acc) when N > 0 ->
    factorial(N-1, N*Acc);
factorial(0, Acc) ->
    Acc.

In foregoing example you can see guards, tail-recursion, usage of accumulators, functions with different arities. Experiment with it.

In my previous post I mentioned that if you use Emacs and you’re editing Erlang source code you can easily comment whole marked region by invoking M-; to which by default bound interactive Lisp function comment-dwim. This works in different modes and it’s very handy especially if your language of choice has no multiline comments.

So far so good, but I must admit sometimes I get the blues when I comment some nested code blocks with M-; keystrokes. Well, maybe I’m blowing things out of proportion about my sadness, but in any case recently I came across really nice small blog post called comment-style and since then I’m happy with comments produced in nested blocks. Let’s see why.

When you comment marked regions in Emacs buffer with M-; the comment-style variable defines the way regions are commented. Default value is plain.

Buffer with Python code before commenting:

def binsearch(seq, key, start, end):
    if end < start:
        return -1
    mid = (start + end) / 2
    if key == seq[mid]:
        return mid
    if key < seq[mid]:
        return binsearch(seq, key, start, mid-1)
    if key > seq[mid]:
        return binsearch(seq, key, start+1, end)

Buffer with Python code after commenting:

def binsearch(seq, key, start, end):
    if end < start:
        return -1
    mid = (start + end) / 2
    # if key == seq[mid]:
#         return mid
    if key < seq[mid]:
        return binsearch(seq, key, start, mid-1)
    if key > seq[mid]:
        return binsearch(seq, key, start+1, end)

As you see visually elements of commented block are not on the same line.

Example with Erlang code before commenting (somewhat far-fetched example though our main concern are comments, not code here):

all(Pred, [Hd|Tail]) ->
    case Pred(Hd) of
        true -> all(Pred, Tail);
        false ->
            io:format("Hd = ~p~n", [Hd]),
            true,
            false
    end;
all(Pred, []) when is_function(Pred, 1) -> true.

After commenting:

all(Pred, [Hd|Tail]) ->
    case Pred(Hd) of
        true -> all(Pred, Tail);
        false ->
            %% io:format("Hd = ~p~n", [Hd]),
%%             true,
            false
    end;
all(Pred, []) when is_function(Pred, 1) -> true.

So the visual issue with comments is the same.

After reading foregoing blog post about comment-style I’ve added this line to my .emacs:

(setq comment-style 'indent)

Easy as pie and which means (taken from Emacs help): ” …`comment-start’ markers should not be put at the left margin but at the current indentation of the region to comment.”

After that addition foregoing commented block in Python mode will look like:

def binsearch(seq, key, start, end):
    if end < start:
        return -1
    mid = (start + end) / 2
    # if key == seq[mid]:
    #     return mid
    if key < seq[mid]:
        return binsearch(seq, key, start, mid-1)
    if key > seq[mid]:
        return binsearch(seq, key, start+1, end)

in Erlang mode:

all(Pred, [Hd|Tail]) ->
    case Pred(Hd) of
        true -> all(Pred, Tail);
        false ->
            %% io:format("Hd = ~p~n", [Hd]),
            %% true,
            false
    end;
all(Pred, []) when is_function(Pred, 1) -> true.

Now I am happy. Thank you, Edward for the tip.

In this short tutorial we will take a look at comparison operations, some arithmetic operations and modules. You may also want to skim over Inro and Part I.

Comparison operations

 Python   Erlang   Description             Erlang Example
 --------+--------+-----------------------+----------------
  <        <        strictly less than
 --------+--------+-----------------------+----------------
  <=       =<       less than or equal
 --------+--------+-----------------------+----------------
  >        >        strictly greater than
 --------+--------+-----------------------+----------------
  >=       >=       greater than or equal
 --------+--------+-----------------------+----------------
  !=       /=       not equal
 --------+--------+-----------------------+----------------
  ==       ==       equal                   1> 1 == 1.
                                            true
                                            2> 1 == 1.0.
                                            true
 --------+--------+-----------------------+----------------
           =:=      exactly equal to        1> 1 =:= 1.
                                            true
                                            2> 1 =:= 1.0.
                                            false
 --------+--------+-----------------------+----------------
           =/=      exactly not equal to
 --------+--------+-----------------------+----------------
  is                object identity
 --------+--------+-----------------------+----------------
  is not            negated obj identity

Python notes

Above comparison operations are supported by all objects. In Python you can chain comparisons:

>>> x, y, z = 1, 3, 7
>>> x < y <= z
True
>>> x < y and y <= z
True

Foregoing examples are identical, except that in second case y is evaluated twice.

Erlang notes

Following order is defined:

number < atom < reference < fun < port < pid < tuple < list < binary

1> 5 < erlang.
true
2> erlang < make_ref().
true

Both Erlang comparison operators (<, =<, >, >=, /=, ==) and Python comparison operators(<, <=, >, >=, !=, ==) make type coerce, “narrower” type is widened to that of another, ie when comparing integer and float first integer is converted to float, etc.
Erlang has special operators without type coerce though: =:= and =/=

Arithmetic operations

I’ll provide only several operations, for more take a look at corresponding language references.

 Python   Python Desc.         Erlang    Erlang Desc.   Example
 --------+--------------------+---------+--------------+--------------------
  x % y    remainder            X rem Y   integer        >>> 7 % 3
           of x / y                       remainder      1
                                          of X / Y       >>> 7.0 % 3
                                                         1.0

                                                         1> 7 rem 3.
                                                         1
                                                         1> 7.0 rem 3.
                                                         =ERROR REP..
 --------+--------------------+---------+--------------+--------------------
  x / y    quotient             X / Y     floating       >>> 7 / 3
           of x and y                     point          2
                                          division       >>> 7.0 / 3
                                                         2.3333333333333335

                                                         1> 7 / 3.
                                                         2.33333
 --------+--------------------+---------+--------------+--------------------
  x // y   (floored) quotient   X div Y   integer        >>> 7 // 3
           of x and y,                    division       2
           integer division                              >>> 7.5 // 3
           (result type is                               2.0
           not forced to be
           int)                                          1> 7 div 3.
                                                         2

Modules

Code in Erlang is organized into units called modules, which is familiar word for Python programmer.

Let’s define sample module with function declaration stored in file mymath.erl :

-module(mymath).
-export([fact/1]).

fact(0) -> 1;
fact(N) -> N * fact(N-1).

What we see here:

1. module’s source code is stored in file with .erl extension
2. module consists of attributes and function declarations which are terminated by period (.)
3. we should provide module declaration defining name of the module

• module name should be atom
• module name is the same as file name minus .erl extension
• module declaration is mandatory
• module declaration attribute should be defined first.

To make functions defined in module accessible outside the module we need to export them, for this -export module attribute exists. We write exported functions inside square brackets in form of func/N where N is the number of arguments of function, called arity. I’ll repeat that functions not pointed in -export will not be accessible outside module.

Before code can be run, module must be compiled. Resulting compiled file will contain extension .beam

If you use Emacs you can compile module with C-c C-k and see results in erlang shell:

1> c("/home/alienoid/dev/erlang/mymath", [{outdir, "/home/alienoid/dev/erlang/"}]).
{ok,mymath}

Or you can compile it directly in shell. Make sure your shell’s current directory is where your mymath.erl lives, if it’s not the case use cd command in erlang shell, cd(“/path/to/dir/with/mymath.erl”) :

1> c(mymath).
{ok,mymath}

To invoke our function fact we use syntax mod:func :

2> mymath:fact(4).
24

Erlang has also -import attribute which allows to import functions into modules, so that you don’t need to use fully-qualified name mod:func to invoke function, again familiar behaviour and naming to Python programmer.

Erlang allows to insert code from file as-is with -include attribute at point where -import is defined, this is used to include records and macro definitions, for example.

Comments in Erlang module begin with character “%“, continue up to end-of-line and may be placed anywhere except inside string and quoted atoms.
Like in Python Erlang has no multiline comments.
If you use Emacs you can easily comment whole region with M-; command after you marked it.

-module(mymath).
-export([fact/1, print_double/1]).
-import(lists, [foreach/2]).
-include("my_records.hrl").

%% sum(L) ->
%%     sum(L, 0).

%% sum([H|T], Acc) ->
%%     sum(T, H+Acc);
%% sum([], Acc) -> Acc.

fact(0) -> 1;
fact(N) -> N * fact(N-1).
print_double(L) ->
    foreach(fun(X) -> io:format("Double of ~p = ~p~n", [X, X*2]) end, L).

Fin.

Next tutorial will be devoted to thorough exploration of functions in Erlang.

When i was describing notations and representation of numbers in Erlang(and Python) i forgot to mention very cool feature of format function in Common Lisp, with which i got acquainted exploring highly-recommended Practical Common Lisp book.

This feature is ~R directive which knows how to say really big numbers in English words.

Here is the example (I use SBCL + SLIME):

CL-USER> (format nil "~R" 1604872898756737459538)
"one sextillion six hundred four quintillion eight hundred seventy-two
quadrillion eight hundred ninety-eight trillion seven hundred fifty-six
billion seven hundred thirty-seven million four hundred fifty-nine
thousand five hundred thirty-eight"

Let’s skim over data types in Erlang today. Check previous tutorial for introduction.

Numbers

In Erlang there are two types of numeric literals: integers and floats.
In Python there are four of them: plain integers(usually called just integers), long integers, floating point numbers, and imaginary numbers.

In addition to conventional notation Erlang has its own specific notations:
1) $char
Gives ASCII value of char
2) base#value
Produces integer. Base is in range [2,36]

1> $A.
65
2> 2#111.
7
3> 16#1A.
26
4> 5.
5
5> 2.55.
2.55000

In Python we can define octinteger and hexinteger correspondingly with:

>>> 017, 0x1A
(15, 26)

To produce integer from different bases in Python we may use built-in int([x[, radix]]), where radix is in range [2,36]:

>>> int("111", 2), int("1A", 16)
(7, 26)

Atom

An atom is a literal or a constant with name. Atoms should start with small letter or it should be enclosed in single quote(‘) if it contains other characters than alphanumeric characters, _, or @.
Atoms can not have values like variables, they are simply names (constant with name).

1> erlang.
erlang
2> 'hello world'.
'hello world'
3> 'Erlang'.
'Erlang'

In Python there are also atoms in form of identifiers or literals, but no correspondence with Erlang’s constant with name type of atom.

More examples with atoms to come when we get acquainted with other data types
and parts of Erlang.

Binary

From a reference manual:
“A binary is used to store an area of untyped memory.
Binaries are expressed using the bit syntax.”

This data type allows to store large raw chunks of memory in an efficient way, much more space-efficient than in lists or tuples.

Binaries are written in double less-than and greater-than brackets and printed in that form as well.
They can be constructed from set of constants or string literal:

1> <<1,2,3>>.
<<1,2,3>>
2> <<"erlang">>.
<<"erlang">>

Or from bound variables:

3> A = 1, B = 2, C = 3.
3
4> Bin = <<A, B, C>>.
<<1,2,3>>

Remember pattern matching ?
Binary (Bin for short) can also be used for matching:

5> <<X, Y, Z>> = Bin.
<<1,2,3>>
6> X.
1
7> Y.
2
8> Z.
3

There is more than that, namely bit syntax which allows to pack/unpack sequences of bits in Bin. Actually bit syntax is an extension to pattern matching.
Let’s pack three variables into 16 bit memory area in a variable Mem, then unpack it to another three variables using bit syntax:

1> X = 2.
2
2> Y = 40.
40
3> Z = 30.
30
4> Mem = <<X:3, Y:7, Z:6>>.
<<74,30>>
5> <<X1:3, Y1:7, Z1:6>> = Mem.
<<74,30>>
6> X1.
2
7> Y1.
40
8> Z1.
30

In foregoing output you see that after packing shell prints packed three variables as <<74,30>> which is 16 bits.

In Python 2.5 there is no built-in data type to support binary data, but in Python 3000 binary data is represented by a separate mutable “bytes” data type.

Fun

Fun is an anonymous function, ie without name.

In Python this is infamous lambda, which had many debates whether it should remain in language or not. Personally i like lambda and glad it survived and will continue its life in Python 3000.

But Ok, back to Erlang’s Fun. Here how we can define it in shell:

1> Double = fun(X) -> X * 2 end.
#Fun<erl_eval.6.56006484>
2> Double(4).
8

In contrast to Python Erlang’s Fun can contain statements and is “more advanced” in general:

1> Print_val = fun(X) -> io:format("X = ~p~n", [X]), X*2 end.
#Fun<erl_eval.6.56006484>
2> Print_val(3).
X = 3
6

>>> print_val = lambda x: print 'x = %d' % x; x*2
  File "<stdin>", line 1
    print_val = lambda x: print 'x = %d' % x; x*2
                              ^
SyntaxError: invalid syntax
>>> print_val = lambda x: x*2
>>> print_val(3)
6

More advanced examples of Fun like Funs with several different clauses i’ll show when discussing in details functions in Erlang in upcoming tutorials.

Tuple

It’s a compound data type with fixed number of terms (Term is a piece of data of any data type). Number of elements in tuple is called size of tuple.
Syntax of tuples is:

1> Tuple = {ruslan, 28}.
{ruslan,28}
2> Info = {data, Tuple}.
{data,{ruslan,28}}

In Python tuples are defined with or without enclosing parentheses:

>>> tup = "ruslan", 28
>>> tup
('ruslan', 28)
>>> info = ('data', tup)
>>> info
('data', ('ruslan', 28))

To get number of elements in tuple.

Erlang:

3> size(Info).
2

Python:

>>> len(info)
2

List

List is a compound data type with variable number of terms. Number of elements in a list is called length of the list.

List is declared in square brackets like in Python.

1> List = [1, 2, 3, "erlang"].
[1,2,3,"erlang"]

Now Erlang specific part, which may be familiar to you if you know a bit Lisp and its CAR/CDR stuff:
List is either an empty list [] or may be represented as head (first element – CAR in Lisp) and tail (remainder of list – CDR in Lisp) which is also a list and often you can see it in form like [H|T].
This representation is recursive:
[]
]
[b | ]]
[a | [b | ]]]
are all lists actually.

Let’s define list in shell:

1> List = [a, 1, {b, 2}].
[a,1,{b,2}]

and unpack it with pattern matching to head and tail:

2> [H|T] = List.
[a,1,{b,2}]
3> H.
a
4> T.
[1,{b,2}]

As you see T(tail) above is also a list.

We can also construct list using head and tail syntax:

5> List2 = .

Working with Erlang you’ll see/use this syntax quite often.

To get number of elements in list.

Erlang:

6> length(List2).
3

Python:

>>> len([1, 2, 3, "python"])
4

String

In Python string type is immutable sequence of characters. In Python there is no separate character type, a character is represented by a string of one item. String literals are written in single or double quotes:

>>> 'xyz', "xyz"
('xyz', 'xyz')

In Erlang strings are enclosed in double quotes, but unlike Python string is not separate data type actually. String “erlang” is just shorthand for list [$e, $r, $l, $a, $n, $g].

Adjacent string literals are concatenated at compile time:

1> "hello" ", world" " in " "erlang".
"hello, world in erlang"

In Python behaviour with adjacent strings is the same:

>>> 'hello' ', world' ' in ' 'python'
'hello, world in python'

Boolean

Again unlike Python Erlang has no built-in Boolean type, but it uses atoms true and false to denote Boolean values:

1> 1 < 2.
true
2> true or false.
true

In Python Boolean is separate data type:

>>> type(True), type(False)
(<type 'bool'>, <type 'bool'>)
>>> 1 < 2
True
>>> True or False
True

Record

Record is a data structure to hold fixed number of elements and it provides method to associate a name with particular element in tuple. It resembles structure in C. Record is not true data type, it is “tuple in disguise”(TM).
It’s syntax is:
-record(Name, {key1=Default1, key2, …}).

But in shell we can not use -record syntax and we do not yet know about modules where it can be defined, so we’ll use ‘rd’ shell command to define record:

1> rd(info, {name="ruslan", age=28, height, weight}).
info

Now let’s create instance of record:

2> X = #info{}.
#info{name = "ruslan",age = 28,height = undefined,weight = undefined}
3> X.
#info{name = "ruslan",age = 28,height = undefined,weight = undefined}

As you see height and weight got “undefined” value.

To extract fields of record we use pattern matching:

4> #info{name=Name, age=Age} = X.
#info{name = "ruslan",age = 28,height = undefined,weight = undefined}
5> Name.
"ruslan"
6> Age.
28

We can also access field of record with “dot syntax”:

7> X#info.name.
"ruslan"

Erlang has more data types like Pid, Port identifier, Reference which will be described later in more advanced topics.There is no direct correspondence to Python’s built-in data types like dictionaries and sets, but Erlang has modules which provide the same semantics.

That’s it for today.

I assume you have already installed Erlang on your system and are able to run erlang shell either from command line or from within Emacs (C-c C-z).

This is a view of Eshell from my Emacs:

Erlang (BEAM) emulator version 5.5.2  [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.5.2  (abort with ^G)
1> 

Let’s do a simple thing and try to assign a value to a variable in the Eshell. First you should know that variables in Erlang start with uppercase letter or underscore (_), for example:

• X
• Name
• Address
• _
• _NoWarn

Ok, let’s finally assign some value to a variable X:

1> X = 4.
4
2>

So far, so good. Make sure to type final “.” at the end of expression when you are done entering code.

Let’s assign new value to X:

2> X = 5.

=ERROR REPORT==== 27-Aug-2007::00:23:02 ===
Error in process <0.29.0> with exit value: {{badmatch,5},[{erl_eval,expr,3}]}

** exited: {{badmatch,5},[{erl_eval,expr,3}]} **
3> 

Here you need to understand and remember couple of things:

1. Erlang uses single assignment, a variable can only be bound once, bar none.
2. In expression X = 5 operator = is not an assignment, but match operator and variables are bound to values using pattern matching mechanism.

I think it won’t be an exaggeration if I say that Pattern Matching permeates Erlang. To understand why we got ERROR REPORT in above example with {badmatch, 5} let’s dive into this pattern matching a bit.

When you use pattern matching’s = operator left side called pattern is matched against right side which is a term(piece of data of any Erlang’s data type: integer, float, list, tuple, etc). If your matching fails you get run-time error, if it succeeds then any unbound variable in lefthand pattern becomes bound.

Returning to our example – first we had an unbound variable:

1> X.
** 1: variable 'X' is unbound **
2> 

Then we bound value to variable X with pattern matching:

2> X = 4.
4
3> X.
4
4> 

Now as the variable X is bound we can’t change its value any more (remember single assignment?) otherwise we’ll get badmatch error like in foregoing X = 5 (4 does not match 5), but we may use = to match our variable:

4> X = 4.
4
5> 

Let’s add a bit of tuples to pattern matching. In Erlang tuple is defined inside {…}. ie:

5> {2, 3}.
{2,3}
6> 

and now pattern matching:

6> {X, Y} = {2, 3}.

=ERROR REPORT==== 27-Aug-2007::01:10:45 ===
Error in process <0.29.0> with exit value: {{badmatch,{2,3}},[{erl_eval,expr,3}]}

** exited: {{badmatch,{2,3}},[{erl_eval,expr,3}]} **
7> {X, Y} = {4, 3}.
{4,3}
8> X.
4
9> Y.
3
10> 

In first attempt we tried to match X with 2 and the matching failed as the value of X is 4, in a second attempt the matching succeeded and unbound variable Y also got its value (became bound).

This may look to you like tuple unpacking in Python:

>>> x, y = (4, 3)
>>> x, y
(4, 3)
>>> x, y = (5, 3)
>>> x, y
(5, 3)

but it’s only visual similarity, though i must admit, useful one.

That’s it for today, later you’ll see how pattern matching is uber-useful and how it is used in different parts of Erlang’s constructs.

Well it’s actually as simple as running M-x man RET module_name RET in minibuffer.

I’m using Erlang setup from rpm on FC7 and by default man pages are located under /usr/lib/erlang/man/ so to use above command in minibuffer you must make sure you have your erlang’s man path either in /etc/man.config or you’ve added it into MANPATH with something like export MANPATH=$MANPATH:/usr/lib/erlang/man in your .bash_profile in case your Erlang man pages are not in location accessible by default.

You may also pass man path in minibuffer manually without changing /etc/man.config or MANPATH. In my case to view man page for ‘lists’ module it will look like:

M-x man RET -M /usr/lib/erlang/man lists RET

I’ve customized some stuff for myself in emacs and made binding to F6 key so that whenever i set cursor on module name in my erlang buffer and press F6 i get man page in another window poped up describing module at point. So if you have in your code or in any buffer, for example:

(em@localhost)2> lists:reverse([1, 2, 3]).

you need to move cursor to lists and press F6. Quite handy if you consult man pages often, which is the case for me now as i learn Erlang.

Here is the corresponding elisp helper function and key binding from my dot emacs:

(defun get-erl-man ()
  (interactive)
  (let* ((man-path "/usr/lib/erlang/man")
         (man-args (format "-M %s %s" man-path (current-word))))
    (man man-args)))

(global-set-key [(f6)] (lambda () (interactive) (get-erl-man)))

Recently I’ve begun to make blog posts and now want to have highlighting of Erlang’s code(actually any code , be it Python, Lisp, etc) in my posts. I’m using The One True Editor and it goes without saying that i want to convert my erlang code into html from within emacs to have corresponding syntax coloring. Htmlize package to the rescue!

After you downloaded package and put

(add-to-list 'load-path "path_to_your_htmlize.el")
(require 'htmlize)

into your dot emacs – you are ready to use it.

M-x htmlize-buffer or M-x htmlize-region will convert whole buffer or selected region with associated decorations into HTML( See C-h a htmlize RET for more functions). By default htmlize will make HTML output in css mode which is not very suitable for inserting into blog post as output includes also style sheets. Excerpt from htmlize documentation:

;; htmlize supports three types of HTML output, selected by setting
;; `htmlize-output-type': `css', `inline-css', and `font'.  In `css'
;; mode, htmlize uses cascading style sheets to specify colors; it
;; generates classes that correspond to Emacs faces and uses <span
;; class=FACE>...</span> to color parts of text.  In this mode, the
;; produced HTML is valid under the 4.01 strict DTD, as confirmed by
;; the W3C validator.  `inline-css' is like `css', except the CSS is
;; put directly in the STYLE attribute of the SPAN element, making it
;; possible to paste the generated HTML to other documents.  In `font'
;; mode, htmlize uses <font color="...">...</font> to colorize HTML,
;; which is not standard-compliant, but works better in older
;; browsers.  `css' mode is the default.

inline-css is the type of output i want. This can be achieved by customizing htmlize-output-type variable with M-x customize htmlize RET or directly modifying dot emacs. Converting:

(add-to-list 'load-path "path_to_your_htmlize.el")
(require 'htmlize)

with htmlize-region and then applying nxml-mode to produced result gives:

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN">
<!-- Created by htmlize-1.34 in inline-css mode. -->
<html>
  <head>
    <title>.emacs</title>
  </head>
  <body style="color: #ffffff; background-color: #000000;">
    <pre>
(add-to-list 'load-path <span style="color: #ffa07a;">"path_to_your_htmlize.el"</span>)
(<span style="color: #00ffff;">require</span> '<span style="color: #7fffd4;">htmlize</span>)
</pre>
  </body>
</html>

As you see I need to cut contents between <pre> </pre> including
<pre> tags itself to insert that then directly into blog post. After that i need manually copy/paste styles from <body> tag into <pre> tag and also add font-size: 8pt to meet my taste. Being lazy i’ve implemented function my-htmlize-region in elisp to make that job for me every time i press F5 key:

(defun my-htmlize-region (beg end)
  "Htmlize region and put into <pre> tag style that is left in <body> tag
plus add font-size: 8pt"
  (interactive "r")
  (let* ((buffer-faces (htmlize-faces-in-buffer))
         (face-map (htmlize-make-face-map (adjoin 'default buffer-faces)))
         (pre-tag (format
                   "<pre style="%s font-size: 8pt">"
                   (mapconcat #'identity (htmlize-css-specs
                                          (gethash 'default face-map)) " ")))
         (htmlized-reg (htmlize-region-for-paste beg end)))
    (switch-to-buffer-other-window "*htmlized output*")
    ; clear buffer
    (kill-region (point-min) (point-max))
    ; set mode to have syntax highlighting
    (nxml-mode)
    (save-excursion
      (insert htmlized-reg))
    (while (re-search-forward "<pre>" nil t)
      (replace-match pre-tag nil nil))
    (goto-char (point-min))))

(global-set-key [(f5)] (lambda (beg end)
                         (interactive "r") (my-htmlize-region beg end)))

So now selecting:

(add-to-list 'load-path "path_to_your_htmlize.el")
(require 'htmlize)

and pressing F5 i get this in another buffer:

<pre style="color: #ffffff; background-color: #000000; font-size: 8pt">
(add-to-list 'load-path <span style="color: #ffa07a;">"path_to_your_htmlize.el"</span>)
(<span style="color: #00ffff;">require</span> '<span style="color: #7fffd4;">htmlize</span>)
</pre>

As you see it’s just what i want:
<pre> tags with content and <pre> tag contains style from above mentioned <body> tag plus font-size is added too.

Factorial in Erlang with syntax coloring as i have it in my Emacs:

-module(factorial).
-export([factorial/1, factorial_rec/1]).

%% tail recursion
factorial(Num) ->
    factorial(Num, 1).

factorial(1, Acc) -> Acc;
factorial(Num, Acc) ->
    factorial(Num-1, Num * Acc).

%% recursion
factorial_rec(1) -> 1;
factorial_rec(Num) ->
    Num * factorial_rec(Num-1).

This is a joy of using Emacs.

In my previous post My Erlang binary search I’ve implemented binary search in Python and Erlang as small exercise for myself after reading Half-baked Ideas blog post. Good article pointing to specifics of implementation of that algorithm in Erlang was posted on Erlane’s blog which is in a nutshell: binary search in Erlang implemented on list data structure is less efficient than linear search.

So to grasp why linear search which is O(N) is faster in Erlang then binary search which is in theory O(logN) go and read that article.
In Python list data structure (I’m talking here about CPython) is implemented as mutable and extensible vector of references to objects(we might say array of pointers), here is the excerpt from listobject.h:

typedef struct {
    PyObject_VAR_HEAD
    /* Vector of pointers to list elements.  list[0] is ob_item[0], etc. */
    PyObject **ob_item;

    /* ob_item contains space for 'allocated' elements.  The number
     * currently in use is ob_size.
     * Invariants:
     *     0 <= ob_size <= allocated
     *     len(list) == ob_size
     *     ob_item == NULL implies ob_size == allocated == 0
     * list.sort() temporarily sets allocated to -1 to detect mutations.
     *
     * Items must normally not be NULL, except during construction when
     * the list is not yet visible outside the function that builds it.
     */
    Py_ssize_t allocated;
} PyListObject;

So Python version works as expected regarding efficiency. While my goal of implementing binary search algorithm in Erlang was learning Erlang and not particulary developing optimized version of that algorithm or using another one it’s always good to pick new tricks of the trade along the way and for me the issue with my implementation of binary search in Erlang just boils down to: your experience and know your tools (language, os, editor(emacs , etc )

As Aldous Huxly said: “Experience is not what happens to you; it’s what you do with what happens to you.”. So make errors (not many in your programs, fix them and gain new experience with your new language.

Another post by Erlane team is good reading describing paradigm shifts when you are newbie and are on your own way to Erlang/OTP wizardry.

I was lurking and watching Erlang quite some time during this year and finally decided give it a shot after Programming Erlang was released. I ordered book in pdf format to rest assured i’ll get it in fastest possible way. After i read book a bit i was hooked

Today i came across binary search in Erlang blog post and decided to implement quickly binary search myself. My first attempt was to do it in python as it’s my daily job’s programming language and it turned this way (no check for edge cases like empty sequence):

def binsearch(seq, key, start, end):
    if end < start:
        return -1
    mid = (start + end) / 2
    if key == seq[mid]:
        return mid
    if key < seq[mid]:
        return binsearch(seq, key, start, mid-1)
    if key > seq[mid]:
        return binsearch(seq, key, start+1, end)

After i looked at code a bit i was surprised i wrote it recursively, earlier i would do it like:

def binsearch(seq, key):
    start = 0
    end = len(seq) - 1

    while True:
        if end < start:
            return -1
        mid = (start + end) / 2
        if key == seq[mid]:
            return mid
        if key < seq[mid]:
            end = mid - 1
        else:
            start = mid + 1

Erlang obviously bends my mind

And my version in Erlang which was just exercise for myself and mainly looks like in original blog post:

-module(search).
-export([binsearch/2]).
-import(lists, [nth/2]).
-include_lib("eunit/include/eunit.hrl").

binsearch(List, Key) ->
    binsearch(List, Key, 1, length(List)).

binsearch(List, Key, LowerBound, UpperBound) ->
    if
        UpperBound < LowerBound -> -1;
        true ->
            Mid = (LowerBound + UpperBound) div 2,
            Item = nth(Mid, List),
            if
                Key < Item ->
                    binsearch(List, Key, LowerBound, Mid-1);
                Key > Item ->
                    binsearch(List, Key, Mid+1, UpperBound);
                true ->
                    Mid
            end
    end.

setup_test_() ->
    {setup,
     fun() -> [7, 11, 14, 40, 50] end,
     fun(L) ->
             [?_assertMatch(3, binsearch(L, 14)),
              ?_assertMatch(-1, binsearch(L, 70))]
     end
    }.

Running unit tests:

(em@localhost)1> search:test().
  All 2 tests successful.
ok