The perils of polyglot programming

Polyglot programming (the practice of knowing and using many programming languages) seems to be all the rage these days. Its adherents claim two benefits:

1. Using the right tool for every job means everything you do is a little bit easier (or better, or faster, or all of the above).
2. Knowing multiple programming paradigm expands your mind and makes you better at programming in every language.

I'm not going to dispute either of these. Well, maybe the second I'll argue with a little: I think you can get most of the benefits by using different paradigms within the same multi-paradigm language, and I'm a bit skeptical of the global benefits (unless you're the type of person who likes writing FORTRAN in Javascript). But I digress, like I said, I think those are both fair claims.

What I don't like is the conclusion that this means you should always use the right tool for the job. What, you the astute reader asks, does this mean you think we should use the wrong tool for the job? No, that would be idiotic, it means I think sometimes using the less optimal tool for the job carries overall benefits.

So what are the dangers of being a polyglot programmers (or the benefits of not being one, if you will)?

Using multiple languages (or any technology) stresses your operations people. It's another piece they have to maintain. If you've got a nice JVM stack, top to bottom, with nice logging and monitoring do you think your ops people really want to hear that they need to duplicate that setup so you can run three Ruby cron jobs? No, they're going to tell you to suck it up and write it up and either see if JRuby works or use Clojure or something, because 1% of your company's code isn't worth doubling their work.

Another risk is that it raises the requirements for all the other developers on the project. Martin Golding said, "Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live." Imagine when you leave your job and the next guy finds out you decided to write some data analysis scripts in APL (for those of you who don't remember, APL is that lovely language that doesn't use ASCII characters). It's fine to use APL if that's something you can require of new hires, it's not fine when your job says "Python developer" (it may actually work for Perl developers, but I assure you it'll be purely coincidental). Learning a new language is hard, learning to write it effectively is harder. Learning a new language for every script you have to maintain is downright painful, and once you know all of them, context switching isn't free for either humans or computers.

I'm not saying write everything in one language, that'd probably leave you writing a lot of code in very suboptimal languages. But choose two or three, not ten. Your ops people will thank you, and so will the guys who have to maintain your code in a decade. At DjangoCon this year Glyph Lefkowitz actually went farther, he argued that not just the code you write, but your entire technology stack should be in one language. But that's a separate discussion, you should watch the video though.

Also, because I'm a big fan of The West Wing, I'd be remiss if I used the word polyglot this many times without linking to a great scene.

Why del defaultdict()[k] should raise an error

Raymond Hettinger recently asked on twitter what people thought del defaultdict()[k] did for a k that didn't exist in the dict. There are two ways of thinking about this, one is, "it's a defaultdict, there's always a value at a key, so it can never raise a KeyError", the other is, "that only applies to reading a value, this should still raise an error". I initially spent several minutes considering which made more sense, but I eventually came around to the second view, I'm going to explain why.

The Zen of Python says, "Errors should never pass silently." Any Java programmer who's seen NullPointerException knows the result of passing around invalid data, rather than propagating an error. There are two cases for trying to delete a key which doesn't exist in a defaultdict. One is: "this algorithm happens to sometimes produce keys that aren't there, not an issue, ignore it", the other is "my algorithm has a bug, it should always produce valid keys". If you don't raise a KeyError the first case has a single line of nice code, if you do raise an error they have a boring try/ except KeyError thing going on, but no big loss. However, if an error isn't raised and your algorithm should never produce nonexistent keys, you'll be silently missing a large bug in your algorithm, which you'll have to hope to catch later.

The inconvenience of ignoring the KeyError to the programmer with the algorithm that produces nonexistent keys is out weighed by the potential for hiding a nasty bug in the algorithm of the programmer who's code should never produce these. Ignoring an exception is easy, trying to find the bug in your algorithm can be a pain in the ass.

RCOS NumPy Talk

I just delivered a talk at RCOS (the open source center on campus) on some of the work I've been doing this semester on PyPy's implementation of NumPy. You can find the slides here, they're built using html5slides, which is pretty swell (for a tool that makes me write HTML directly that is).

The run-time distinction

At PyCodeConf I had a very interesting discussion with Nick Coghlan which helped me understand something that had long frustrated me with programming languages. Anyone who's ever taught a new programmer Java knows this, but perhaps hasn't understood it for what it is. This thing that I hadn't been appreciating was the distinction some programming languages make between the language that exists at compile time, and the language that exists at run-time.

Take a look at this piece of Java code:

class MyFirstProgram {
    public static void main(String[] args) {
        System.out.println("Hello World!");
    }
}

Most people don't appreciate it, but you're really writing in two programming languages here, one of these languages has things like class and function declarations, and the other has executable statements (and yes, I realize Java has anonymous classes, they don't meaningfully provide anything I'm about to discuss).

Compare that with the (approximately) equivalent Python code:

def main():
    print "Hello World"

if __name__ == "__main__":
    main()

There's a very important thing to note here, we have executable statements at the top level, something that's simply impossible in Java, C, or C++. They make a distinction between the top level and your function's bodies. It follows from this that the function we've defined doesn't have special status by virtue of being at the top level, we could define a function or write a class in any scope. And this is important, because it gives us the ability to express things like decorators (both class and function).

Another example of this distinction that James Tauber pointed out to me is the import statement. In Python is it a line of executable code which invokes machinery in the VM to find a module and load it into the current namespace. In Java it is an indication to the compiler that a certain symbol is in scope, it's never executed.

Why would anyone care about this distinction though? It's clearly possibly to write programs in languages on both ends of the spectrum. It appears to me that the expressiveness of a programming language is really a description of what the distance between the compile time language and the runtime language is. Python stands on one end, with no distinction, whereas C/C++/Java stand on the other, with a grand canyon separating them.

But what about a language in the middle? Much of PyPy's code is written in a language named RPython. It has a fairly interesting property though, its run-time language is pretty close to Java in semantics, it's statically typed (though type inferenced), it's compile time language is Python. In practice this means you get many of the benefits in expressiveness as you do from using Python itself. For example you can write a decorator, or generate a class. A good example of this power is in PyPy's NumPy implementation. We're able to create the code for doing all the operations on different dtypes (NumPy's name for the different datatypes its arrays can represent) dynamically, without needing to resort to code generation or repeating ourselves, we're able to rely on Python as our compile time (or meta-programming) language. The "in-practice" result of this is that writing RPython feels much more like writing Python than it does like writing Java, even though most of your code is written under the same constraints (albeit without the need to explicitly write types).

The distinction between compile-time and run-time in programming languages results in both more pain for programmers, as well as more arcane structures to explain to new users. I believe new languages going forward should make it a goal to either minimize this difference (as Python does) or outfit languages with more powerful compile-time languages which give them the ability to express meta-programming constructs.

So you want to write a fast Python?

Thinking about writing your own Python implementation? Congrats, there are plenty out there [1], but perhaps you have something new to bring to the table. Writing a fast Python is a pretty hard task, and there's a lot of stuff you need to keep in mind, but if you're interested in forging ahead, keep reading!

First, you'll need to write yourself an interpreter. A static compiler for Python doesn't have enough information to do the right things [2] [3], and a multi-stage JIT compiler is probably more trouble than it's worth [4]. It doesn't need to be super fast, but it should be within 2x of CPython or so, or you'll have lost too much ground to make up later. You'll probably need to write yourself a garbage collector as well, it should probably be a nice, generational collector [5].

Next you'll need implementations for all the builtins. Be careful here! You need to be every bit as good as CPython's algorithms if you want to stand a chance, this means things like list.sort() keeping up with Timsort [6], str.__contains__ keeping up with fast search [7], and dict.__getitem__ keeping up with the extremely carefully optimized Python dict [8].

Now you've got the core language, take a bow, most people don't make it nearly this far! However, there's still tons of work to go, for example you need the standard library if you want people to actually use this thing. A lot of the stdlib is in Python, so you can just copy that, but some stuff isn't, for that you'll need to reimplement it yourself (you can "cheat" on a lot of stuff and just write it in Python though, rather than C, or whatever language your interpreter is written in).

At this point you should have yourself a complete Python that's basically a drop-in replacement for CPython, but that's a bit slower. Now it's time for the real work to begin. You need to write a Just in Time compiler, and it needs to be a good one. You'll need a great optimizer that can simultaneously understand some of the high level semantics of Python, as well as the low level nitty gritty of your CPU [9].

If you've gotten this far, you deserve a round of applause, not many projects make it this far. But your Python probably still isn't going to be used by the world, you may execute Python code 10x faster, but the Python community is more demanding than that. If you want people to really use this thing you're going to have to make sure their C extensions run. Sure, CPython's C-API was never designed to be run on other platforms, but you can make it work, even if it's not super fast, it might be enough for some people [10].

Finally, remember that standard library you wrote earlier? Did you make sure to take your time to optimize it? You're probably going to need to take a step back and do that now, sure it's huge, and people use every nook and cranny of it, but if you want to be faster, you need it to be faster too. It won't do to have your bz2 module be slower, tarnishing your beautiful speed results [11].

Still with me? Congratulations, you're in a class of your own. You've got a blazing fast Python, a nicely optimized standard library, and you can run anyone's code, Python or C. If this ballad sounds a little familiar, that's because it is, it's the story of PyPy. If you think this was a fun journey, you can join in. There are ways for Python programmers at every level to help us, such as:

• Contributing to our performance analysis tool, this is actually a web app written using Flask.
• Contribute to speed.pypy.org which is a Django site.
• Provide pure Python versions of your C-extensions, to ensure they run on alternative Pythons.
• Test and benchmark your code on PyPy, let us know if you think we should be faster! (We're always interested in slower code, and we consider it a bug)
• Contribute to PyPy itself, we've got tons of things to do, you could work on the standard library, the JIT compiler, the GC, or anything in between.

Hope to see you soon [12]!

DjangoCon Europe 2011 Slides

I gave a talk at DjangoCon Europe 2011 on using Django and PyPy (with another talk to be delivered tomorrow!), you can get the slides right on bitbucket, and for those who saw the talk, the PyPy compatibility wiki is here.

This Summer

For the past nearly two years I've been an independent contractor working primarily for Eldarion, and it's been fantastic, I can't say enough good things. However, this summer I'm shaking things up a bit and heading west. I'll be an intern at Quora this summer. They're a Python shop, and it's going to be my job to make everything zippy. Specifically I'll be making the site run on PyPy, and then tuning the hell out of both their codebase and PyPy itself. I'm super excited about the chance to get a major site running on PyPy, and to contribute as many performance improvements upstream as possible.

My experience with the computer language shootout

For a long time we, the PyPy developers, have known the Python implementations on the Computer Language Shootout were not optimal under PyPy, and in fact had been ruthlessly microoptimized for CPython, to the detriment of PyPy. But we didn't really care or do anything about it, because we figured those weren't really representative of what people like to do with Python, and who really cares what it says, CPython is over 30 times slower than C, and people use it just the same. However, I've recently have a number of discussions about language implementation speed and people tend to cite the language shootout as the definitive source for cross-language comparisons. So I decided to see what I could do about making it faster.

The first benchmark I took a stab at was reverse-complement, PyPy was doing crappily on it, and it was super obviously optimized for CPython: every loop possible was pushed down into functions known to be implemented in C, various memory allocation tricks are played (e.g. del some_list[:] removes the contents of the list, but doesn't deallocate the memory), and bound method allocation is pulled out of loops. The first one is the most important for PyPy, on PyPy your objective is generally to make sure your hot loops are in Python, the exact opposite of what you want on CPython. So I started coding up my own version, optimized for PyPy, I spent some time with our debugging and profiling tools, and whipped up a nice implementation that was something like 3x faster than the current one on PyPy, which you can see here. Generally the objective here was to make sure the program does as little memory allocation in the hot loops as possible, all of which are in Python. Try that with your average interpreter.

So I went ahead and submitted it, thinking PyPy would be looking 3 times better when I woke up. Naturally I wake up to an email from the shootout, which says that I should provide a Python 3 implementation, and that it doesn't work on CPython. What the hell? I try to run it myself and indeed it doesn't. It turns out on CPython sys.stdout.write(buffer(array.array("c"), 0, idx)) raises an exception. Which is a tad unfortunate because it should be an easy way to print out part of an array of characters without needing to allocate memory. After speaking with some CPython core developers, it appears that it is indeed a bug in CPython. And I noticed on PyPy buffer objects aren't nearly as efficient as they should be, so I set out in search of a new way to work on CPython and PyPy, and be faster if possible. I happened to stuble across the method array.buffer_info which returns a tuple of the memory address of the array's internal storage and its length, and a brilliant hack occurred to me: use ctypes to call libc's write() function. I coded it up, and indeed it worked on PyPy and CPython and was 40% faster on PyPy to boot. Fantastic I thought, I'll just submit this and PyPy will look rocking! Only 3.5x slower than C, not bad for an interpreter, in a language that is notoriously hard to optimize. You can see the implementation right here, it contains a few other performance tricks as well, but nothing too exciting.

So I submitted this, thinking, "Aha! I've done it". Shortly, I had an email saying this has been accepted as an "interesting alternative" because it used ctypes, which is to say it won't be included in the cumulative timings for each implementation, nor will it be listed with the normal implementations for the per-benchmark scores. Well crap, that's no good, the whole point of this was to look good, what's the point if no one is going to see this glorious work. So I sent a message asking why this implementation was considered alternative, since it appeared fairly legitimate. I received a confusing message questioning why this optimization was necessary, followed by a suggestion that perhaps PyPy wasn't compatible enough with (with what I dare not ask, but the answer obviously isn't Python the abstract language, since CPython had the bug!).

Overall it was a pretty crappy experience. The language shootout appears to be governed by arbitrary rules. For example the C implementations use GCC builtins, which are not part of the C standard, making them not implementation portable. The CPython pidigits version uses a C extension which is obviously not implementation portable (by comparison every major Python implementation includes ctypes, only CPython, and to varying extents IronPython and PyPy, support the CPython C-API), although here PyPy was allowed to use ctypes. It's also not possible to send any messages once your ticket has been marked as closed, meaning to dispute a decision you basically need to pray the maintainer reopens it for some reason. The full back and forth is available here. I'm still interested in improving the PyPy submissions there (and further optimizing PyPy where needed). However given the seemingly capricious and painful submission process I'm not really inclined to continue work, nor can I take the shootout seriously as an honest comparison of languages.

PyPy San Francisco Tour Recap

Yesterday was the last day of my (our) tour of the Bay Area, so I figured I'd post a recap of all the cool stuff that happened.

Django and Python 3 (Take 2)

About 18 months ago I blogged about Django and Python 3, I gave the official roadmap. Not a lot has changed since then unfortunately, we've dropped 2.3 entirely, 2.4 is on it's way out, but there's no 3.X support in core. One fun thing has changed though: I get to help set the roadmap, which is a) cool, b) means the fact that I care about Python 3 counts for something. I'm not going to get into "Why Py3k", because that's been done to death, but it's a better language, that I'd rather program in, so the only question is how do we get there.

I posted a while ago on Hacker News that I foresaw Python 3 support by the end of this summer. I still think that's reasonable, but there's a tiny unspoken question there: how do we go from 0 to there. I have an answer! Drum roll please... it turns out, as a student, I tend to have a lot of free time over the summer, and Google has this lovely thing called Google Summer of Code, where they give students money to work on open source. So I'd like to make Python 3 support for Django a GSOC project this summer. With myself either mentoring or student-ing. I don't care which role I take, just that someone who's committed to the project takes it on. I think this task is eminently reasonable for the timeline, especially in light of the work done by Martin von Löwis which, even though it probably doesn't apply, lays a lot of the ground work and identifies a lot of the key issues (and features a lot of utilities which will probably be of use).

All that being said any statements here or elsewhere (especially one involving timelines) reflects my personal goals and beliefs, not necessarily any other Django core developers, and certainly not an official project position.

PyCon 2011 is going to be Awesome

If you hang out with Pythonistas you've probably already heard, but if you haven't, come to PyCon! It's going to be awesome. Here are just a few reasons why:

• The talks will be great, I've got 2 that I'm super excited about, but here are five more I wouldn't miss for the world:

  • "Dude, Where's My RAM?" - A deep dive into how Python uses memory
  • Writing great documentation
  • Panel: Python in Schools: Teaching It and Teaching With It.
  • An (biased) survey of the python web
  • Reverse-engineering Ian Bicking's brain: inside pip and virtualenv
  • Jython Concurrency

• The hallway track will be amazing. This is something that not enough people realize at their first PyCon: you don't have to attend every talk, if you get into a great discussion with some other developers between talks just skip the next few, we record the talks so you can have great conversations.

• The sprints will be incredible. I've missed the sprints at the last two PyCons, and I'm beyond excited to be able to stay for them this year. The sprints are a great chance to get involved in an open source project, or just dedicate some time to contributing to one. Going to them will make you a better developer, you'll learn the codebase better and have experienced devs who want nothing more than to give you feedback.

You've got until January 25th to secure the early bird pricing for PyCon tickets, and it is expected to sell-out this year, so get registered!

Announcing VCS Translator

For the past month or so I've been using a combination of Google, Stackoverflow, and bugging people on IRC to muddle my way through using various VCS that I'm not very familiar with. And all too often my queries are of the form of "how do I do git foobar -q in mercurial?". A while ago I tweeted that someone should write a VCS translator website. Nobody else did, so when I woke up far too early today I decided I was going to get something online to solve this problem, today! About 6 hours later I tweeted the launch of VCS translator.

This is probably not even a minimum viable product. It doesn't handle a huge range of cases, or version control systems. However, it is open source and it provides a framework for answering these questions. If you're interested I'd encourage you to fork it on github and help me out in fixing some of the most requested translation (I remove them once they're implemented).

My future goals for this are to allow commenting, so users can explain the caveats of the translations (very infrequently are the translations one-to-one) and to add a proper API. Moreover my goal is to make this a useful tool to other programmers who, like myself, have far too many VCS in their lives.

2010 in Review

Now that the decades almost over (or maybe not, if the millennium started in 2001 did the decade start then as well?), it's time for a little year in review:

• Finished another year of school, I can see the light at the end of the tunnel.
• Went to PyCon, attended the language summit, gave a talk, had a blast.
• Got an iMac.
• Went to DjangoCon.eu, gave a talk, had a blast.
• Went to PyOhio, gave a talk, had a blast (starting to sense a trend?).
• Did GSOC again, this time working on a refactor of some of the ORM infrastructure for Django with an eye towards adding support for non-relational databases. It went pretty well, haven't started the merging yet though.
• Went to a conference on education, gave a talk, had a blast.
• Went to DjangoCon, wait for it... gave a talk, had a blast.
• Turned 20.
• Became a Django core developer.
• Increased my contributions to PyPy.
• Participated in the PyCon program committee.
• Said thank you to a few people who were absurdly deserving of it.
• Utterly failed at my attempt to blog once a week for a year, and then failed again at National Blog Post Month (aka NAMBLA).

I'm sure I'm forgetting tons of stuff, that's ok, it was a pretty good year either way. No predictions for 2011, just a hope that it'll kick some ass.

Getting the most out of tox

tox is a recent Python testing tool, by Holger Krekel. It's stated purpose is to make testing Python projects against multiple versions of Python (or different interpreters, like PyPy and Jython) much easier. However, it can be used for so much more. Yesterday I set it up for django-taggit, and it's an absolute dream, it automates testing against four different versions of Python, two different versions of Django, and it automates building the docs and checking for any warnings from Sphinx. I'll try to give a run through on what exactly you need to do to set this up with your project.

First create a tox.ini at the root of your project (i.e. in the same directory as your setup.py). Next create a [tox] section, and list out the enviroments you'd like to be tested (i.e. which Pythons):

[tox]
envlist =
    py25, py26 , py27, pypy

The enviroments we've listed out are a few of the ones included with tox, they point at specific versions of Python and use the default testing setup. Now add a [testenv] section which will tell tox how to actually run your tests:

[testenv]
commands =
    python setup.py test
deps =
    django==1.2.3

commands is the list of commands tox will run, and deps specifies any dependencies that are needed to run the tests (tox creates a virtualenv for each enviroment and doesn't include system wide site-packages, so you need to make sure you list everything needed by default here). If you want to use this same python setup.py test formulation you'll need to be using setuptools or distribute for your setup.py and provide the test_suite argument, Eric Holscher provides a good run down for how to do this for Django projects.

Now you should be able to just type tox into your command line and it will try to run your tests in each of the enviroments you specified. Hopefully they're all passing (future test runs will go faster, for the first run it has to install all the dependencies). The next thing you may want to do is get it setup to build your documentation. To do this create a [testenv:docs] section:

[testenv:docs]
changedir = docs
deps =
    sphinx
commands =
    sphinx-build -W -b html -d {envtmpdir}/doctrees . {envtmpdir}/html

This tells tox a few things. First changedir tells it that to run these commands it should cd into the docs/ directory (if you're docs live elsewhere, change as appropriate). Next it has sphinx as a dependency. Finally the commands invoke sphinx-build, -W makes warnings into errors (so you get an approrpiate failure status code), -b html uses the HTML builder, and the rest of the parameters tell Sphinx where the docs live and to put the output in the temporary directory that tox creates for each env.

Now all you need to do is add docs to the envlist, and a tox run will build your documentation.

The last thing you might want to do is set it up to test against multiple versions of a package (such as Django 1.1, Django 1.2, and trunk). To do this create another section whose name includes both the Python version and dependency version, e.g. [testenv:py25-trunk]. In it place:

[testenv:py25-trunk]
basepython = python2.5
deps =
    http://www.djangoproject.com/download/1.3-alpha-1/tarball/

This "inherits" from the default testenv, so it still has its commands, but we specify the basepython indicating this testenv is for python 2.5, and a different set of dependencies, here we're using the Django 1.3 alpha. You'll need to do a bit of copy-paste and create one of these for each version of Python you're testing against, and make sure to add each of these to the envlist.

At this point you should have a lean, mean, testing setup. With one command you can test your package with different dependencies, different pythons, and build your documentation. The tox documentation features tons of examples so you should use it as a reference.

Programming Languages Terminology

This semester I've been taking a course titled "Programming Languages". Since I'm a big languages and compilers nerd this should be wonderful. Unfortunately every time I'm in this class I'm reminded of just what a cluster-fuck programming language terminology is. What follows are my definitions of a number of terms:

• Dynamic typing (vs static typing): Values do not have a type until runtime. Has nothing to do with the declaration of types (i.e. a type inferenced language is not dynamically typed).
• Type safety (vs type unsafety): Operations cannot be performed on values which do not support them, these operations need not be prohibited prior to execution, a run time exception suffices.
• Strong typing (vs weak typing): Implicit conversions are not performed. This has nothing to do with static or dynamic typing. Rather it references to whether a language will perform an operation such as '1' + 2. For example Python raises a TypeError here, whereas PHP returns 3. This one is slightly muddied by the fact that in languages with user defined types (and the ability to implement behaviors on operators), a type can really do anything it likes, thus this is less of an aspect of the core language and more one of the included types and functions.

There are probably some terms I've missed, but for these terms I think my definitions roughly match the consensus on them.