Urlencode and urldecode from a command line

June 2, 2010

Recently I have worked with a REST Web service and had to URL encode a lot of values from the command line.

If all you need is to URL encode data for a POST request you can use curl with –data-urlencode parameter, but I needed to encode values that went into a URL query string.
So for the task I wrote a short code in Python to do just that (there are examples available in BASH or Perl, but this is a pure Python using only standard library):

python -c "import sys, urllib as ul; print ul.quote_plus(sys.argv[1])"

Assign it to an alias by putting single quotes around the expression and you’re good to go:

$ alias urlencode='python -c "import sys, urllib as ul; print ul.quote_plus(sys.argv[1])"'

Example:

$ urlencode 'q werty=/;'
q+werty%3D%2F%3B

And the decode part:

python -c "import sys, urllib as ul; print ul.unquote_plus(sys.argv[1])"
$ alias urldecode='python -c "import sys, urllib as ul; print ul.unquote_plus(sys.argv[1])"'
$ urldecode 'q+werty%3D%2F%3B'
q werty=/;

Review of Grok 1.0 Web Development

April 12, 2010

It took me a while to finish reading the book but here we go.

I wasn’t disappointed by the book and absolutely sure that it delivers on its promise to show that Grok can be an excellent fit for many kinds of Web development projects.

I think I’ll have to finish my Grok based blog now :)

Grok is largely based on Zope Tookit and that’s a very powerful set of libraries developed over many years that provides support for forms, full text search, authentication, i18n, security, components, persistence, testing, and many more.

Main distinguishing Grok concepts emphasized by the book:

  • component architecture
  • object database (ZODB)
  • object publishing and traversal
  • convention over configuration and do not repeat yourself (DRY) principle

The book introduces a simple “to-do” application to the readers. Gradually the application grows with new features like storing data in object database, forms, full text catalog search, security, component architecture.

There is a separate chapter devoted to integration with relational databases. This is especially important topic for Grok because traditionally Zope and Grok were tightly bound to ZODB.

All in all it’s been a pleasant read and I think it’s a great introduction to Grok web framework.

P.S. I may sound very optimistic regarding Grok and Zope, but I can’t do anything about it – I have a soft spot for Grok, it’s an excellent piece of software :)

Enjoy it!


Preview of Grok 1.0 Web Development by Carlos de la Guardia

February 26, 2010

I’ve been asked by PACKT publishing to review a new book  Grok 1.0 Web Development.

I gladly accepted their proposal. For those interested – yes, they sent me a free PDF version of the book, but I would buy the book anyway because it is about the project I love and associate myself with even though I’ve not been actively participating in it recently.

Personally I think that the release of this book is a major event for the whole Grok community. Having comprehensive documentation for people just starting with a smashing Web framework Grok is of utter importance.

What’s interesting is that I’ve been thinking about writing a book on Grok myself, but Carlos de la Guardia beat me to the punch and after skimming through the book I’m glad he did :)

What I liked instantaneously about the book is the presence of a chapter on integration with relational databases, the topic that in the Zope world was kind of obscure in the past.

When I finish reading the book I’ll post a full review so stay tuned and meanwhile you can take a sneak peak at the book by reading a freely available Chapter No 5: Forms



My first crack at Nokia N900 Maemo Python development

December 28, 2009

Got my shiny new Nokia N900 – to say that I’m happy is an understatement :)

For me as a long standing Linux user and Pythonista N900 has an unassailable advantage – its Maemo platform is Linux based and I can program in Python for the platform.

So it’s time to get my feet wet and learn some Maemo programming. I decided to take a stab at writing a small Anki client using Python and Qt. As a result I have a working but still very raw prototype called KTAnki that successfully runs on my N900.

For KTAnki I used PyQt4 library, though I should take a closer look at Nokia’s own bindings – PySide.

Here are some resources that I found helpful when prototyping my first Maemo PyQt application:

  1. N900 USB Networking
  2. PyMaemo QuickStart Guide
  3. Getting started with PyQt for Maemo
  4. PyQt Application Development On Maemo
  5. Hildonized applications for Maemo 5 with Qt4.5.3 and Qt4.6
  6. Rapid GUI Programming with Python and Qt
  7. Qt Reference Documentation

CSSBuilder for Data URIs

November 30, 2009

I’ve added a new repository for CSSBuilder at http://github.com/rspivak/cssbuilder

CSSBuilder is a command line utility and library written in Python for embedding Data URIs into CSS files.

You can find more information about Data URI at:

http://en.wikipedia.org/wiki/Data_URI_scheme

http://www.nczonline.net/blog/2009/10/27/data-uris-explained/

http://www.phpied.com/data-urls-what-are-they-and-how-to-use/

 


Rebinding variable outside of the local scope

October 7, 2009

The other day I had to explain why in Python in contrast to Perl/Lisp/Scheme one can’t rebind variable outside of the local scope besides the global scope, i.e. the following function will result in an UnboundLocalError exception in Python 2.x

def outer():
    count = 0
    def inner():
        count += 1 # rebinding
    inner()
    print count
>>> outer()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "scope.py", line 8, in outer
    inner()
  File "scope.py", line 6, in inner
    count += 1 # rebinding
UnboundLocalError: local variable 'count' referenced before assignment

There are several workarounds.

For one thing it is possible to use a variable holding a mutable object:

def outer():
    ns = dict(count=0)
    def inner():
        ns['count'] += 1 # rebinding
    inner()
    print ns['count']
>>> outer()
1

Another approach is to use a function object itself to store the variable:

def outer():
    outer.count = 0
    def inner():
        outer.count += 1 # rebinding
    inner()
    print outer.count
>>> outer()
1

The good news is that there is no need for workarounds anymore in Python 3 which introduced nonlocal keyword:

def outer():
    count = 0
    def inner():
        nonlocal count
        count += 1 # rebinding
    inner()
    print(count)
>>> outer()
1

IPython profile for Grok

March 30, 2009

To use IPython shell from Grok based project is easy, just add ipython to eggs directive in your buildout.cfg

My part with ipython added looks like this:

[app]
recipe = zc.recipe.egg
eggs = grok-awesome
       ipython
       Paste
       PasteScript
       PasteDeploy
interpreter = python-console

But I wanted to be able to inspect Grok instance and use path TAB auto-completion to navigate ZODB hierarchy.

Inspired by existing IPython profile fo Zope2 I whipped out basic implementation for Grok.

It’s available under grok-ipython. Now all IPython power can be used when inspecting objects in Grok instance:

(mygrok)[alienoid@capricorn grok-awesome]$ bin/ipython -p grok
IPython shell for Grok.

Bound object names:
-------------------
  root
  ctx

Bound command names:
--------------------
  cdg / ;cdg
  lsg / ;lsg
  pwdg
  sync
  commit

Grok IPython 0.9.1   Py 2.5.2

In [1]: root
Out[1]: <zope.app.folder.folder.Folder object at 0x9003e6c>

In [2]: ctx
Out[2]: <zope.app.folder.folder.Folder object at 0x9003e6c>

In [3]: ;cdg blog
------> cdg("blog")
Out[3]: u'/blog'

In [4]: ctx
Out[4]: <grokawesome.blog.Blog object at 0x81d2d6c>

In [5]: ;cdg 
my-first-post  second-post    

In [5]: ;cdg my-first-post
------> cdg("my-first-post")
Out[5]: u'/blog/my-first-post'

In [6]: ctx
Out[6]: <grokawesome.entry.BlogEntry object at 0xb7d556ac>

In [7]: pwdg
Out[7]: u'/blog/my-first-post'

In [8]: ctx?
Type:           BlogEntry
Base Class:     <class 'grokawesome.entry.BlogEntry'>
String Form:    <grokawesome.entry.BlogEntry object at 0xb7d556ac>
Namespace:      Interactive
Length:         0
File:           /home/alienoid/mygrok/grok-awesome/src/grokawesome/entry.py
Docstring:
    <no docstring>


In [9]: ctx??
Type:             BlogEntry
Base Class:       <class 'grokawesome.entry.BlogEntry'>
String Form:   <grokawesome.entry.BlogEntry object at 0xb7d556ac>
Namespace:        Interactive
Length:           0
File:             /home/alienoid/mygrok/grok-awesome/src/grokawesome/entry.py
Source:
class BlogEntry(grok.Container):
    grok.implements(IBlogEntry)

    portal_type = 'blogentry'
    created = property.DCProperty('created')
    modified = property.DCProperty('modified')

    def __init__(self, title, content,
                 summary=u'',
                 categories=None,
                 allow_comments=False):
        super(BlogEntry, self).__init__()
        self.title = title
        self.content = content
        self.summary = '' if summary is None else summary
        self.categories = [] if categories is None else categories
        self.allow_comments = allow_comments

TinyMCE widget in Grok

March 24, 2009

I have added TinyMCE widget to grok-awesome for editing html and it was easy as pie.

If you want TinyMCE in your Grok based project it can be achieved with following steps:

1. Add dependencies to setup.py into install_requires section:

                        'hurry.tinymce',
                        'hurry.zopetinymce',

hurry.tinymce allows you to include TinyMCE into your project without additional burden, you don’t need to do anything. Well, almost – just make sure to call:
from hurry.tinymce import tinymce
tinymce.need()

to trigger inclusion of TinyMCE in the web page.

2. Create custom widget which calls tinymce.need() as noted above and initializes TinyMCE

from zope.app.form.browser import TextAreaWidget
from hurry.tinymce import tinymce

template = """%(widget_html)s
<script type="text/javascript">
  tinyMCE.init({
    mode : "exact",
    elements: "%(elements)s",
    theme: "advanced"
  });
</script>"""

class TinyMCEWidget(TextAreaWidget):
    """Widget to edit html using TinyMCE WYSIWYG editor."""

    def __call__(self, *args, **kw):
        widget_html = super(TinyMCEWidget, self).__call__(*args, **kw)
        tinymce.need()
        return template % {'widget_html': widget_html,
                           'elements': self.name,
                           }

3. In your add/edit form use your custom widget for the field you want and off you go:

form_fields['content'].custom_widget = TinyMCEWidget

Decode hex pairwise with zip

March 20, 2009

The other day I had to decode strings containing hex numbers into plain integers.

I could use built-in int([ x [, radix]]) with radix=16 to convert a string to an integer and forget about it, but I needed to decode pairs from the hex number so that in the end from the string ‘FFEEDD’ I get 255, 238, 221

But wouldn’t you know it, there is a standard solution to get pairs in Python and it’s built-in zip function.

It’s a well known function for experienced developers, but there is one documented feature of it that doesn’t always catch one’s eye and it’s a general form for clustering a data series into n-length groups zip(*[iter(s)]*n) which is described in official docs for zip built-in.

With that in mind solution to my problem boils down to a one-line decoder:

>>> s = 'FFDDEE'
>>> [int(''.join(pair), 16) for pair in zip(*[iter(s)]*2)]
[255, 221, 238]

For those wondering how it works:

1) iter(s) creates iterable from the s

>>> iter(s)
<iterator object at 0x8a3c2cc>

2) [iter(s)] * 2 creates list of 2 iterators that all use the same underlying iterable iter(s). The important thing is that created iterators share the same iterable, so when some data is consumed by first iterator that data becomes unavailable for the second iterator and vice versa.

You can see from the output that iterator objects are located at the same address. If you are new to this behavior you can read on about shallow copies.

>>> [iter(s)]*2
[<iterator object at 0x8a3c2cc>, <iterator object at 0x8a3c2cc>]

3) leading * in zip’s parameter unpacks list from step (2) into two separate parameters for zip function. Basically it becomes zip(iterator, iterator).

More about unpacking argument list you can read in official docs.

>>> zip(*[iter(s)]*2)
[('F', 'F'), ('D', 'D'), ('E', 'E')]

Grok Awesome

March 12, 2009

I’ve started developing blog software for “A Smashing Web Framework” Grok.

The reasons behind that:

  • There is no full-featured blog software for Grok. There is a simple version Grokstar, but as for me it doesn’t fit the bill.
  • I created simple blog for Zope3 several years ago and let it languish. Now it’s time to revive it taking into account how Grok simplifies development with Zope3 technologies.
  • I’m not completely happy with WordPress. For one thing, I’d like to have easy-to-use REST API so I can use it in my Emacs to make posts.
  • I simply love Grok. You should try it and you’ll love it too.

My fledgling project lives on GitHub and it’s called grok-awesome :)