As you may have heard by now, I'm joining Rackspace. This was a very hard decision, I love my job at Rdio, I love my coworkers, and I love the product we're building. However, Rackspace has given me an opportunity to make my work on open source projects (particularly PyPy) a major part of my job. I'll also be spending time on OpenStack, the SDKs, and internal developer evangelism. I'll be staying in San Francisco.
You can find the rest here. There are view comments.
I've been reading Sheryl Sandberg's "Lean In". It's a good book and I recommend it. One passage recently caught my attention. Writing about her first day at a new job: "'I can't believe you've gotten this far, or even how you can understand basic economics, without knowing how to use Lotus.' I went home convinced that I was going to get fired." (p62)
A number of years ago my college roommate and I both had summer internships and we were discussing them. One thing we both noted, while looking at who some of our coworkers would be, was that all of them went to better schools than we did. RPI is not, by any means, a bad school, but our coworkers had gone to Stanford, MIT, Harvard, Berkeley, etc., the best of the best. In light of this, he commented that he was concerned that he might not be qualified.
In retrospect, that's the completely obvious reaction that nearly everyone would have to that situation. It wasn't the one I had. My reaction to seeing that every single coworker went to a better school was: they must have thought I was qualified based on something else, since my school didn't count as evidence in my favor.
Instead of being unnerved by the situation, I was positively reassured. Were I in Sandberg's position, my first thought would probably have been, "Gee, I probably got this far in economics because I kick ass at something other than Lotus Notes".
The lesson I take from this is, whenever humanly possible, try to take a positive, actionable lesson away from a situation or criticism. When that fails, the person you're talking to probably doesn't want to help you, they just want to make you feel bad. And don't ever let anyone make you feel bad about not knowing Lotus.
You can find the rest here. There are view comments.
I had a fairly fun project at work recently, so I thought I'd write about it. We currently store BSON blobs of data in many places. This is unfortunate, because BSON is bloated and slow (an array is internally stored as a dictionary mapping the strings "0", "1", "2", etc. to values). So we wanted to migrate to msgpack, which I've measured as requiring 46% of the space of BSON, and being significantly faster to deserialize (we aren't concerned with serialization speed, though I'm relatively confident that's faster as well).
The one trick we wanted to pull was to do the migration in place, that is gradually rewrite all the columns' data from BSON to msgpack. This is only possible if the data can be interpreted as one or the other unambiguously. So I was tasked if finding out if this was possible.
The first thing that's important to know about BSON is that the first 4-bytes are the length of the entire document (in bytes) as a signed integer, little endian. msgpack has no specific prefix, the first bytes are merely the typecode for whatever the element is. At Rdio, we know something about our data though, because BSON requires all top-level elements to be dictionaries, and we're just re-serializing the same data, we know that all of these msgpacks will have dictionaries as the top level object.
Because a BSON blob starts with its size, in bytes, we're going to try to find the smallest possible 4-byte starting sequence (interpreted as an integer) one of our payloads could have, in order to determine what the smallest possible ambiguity is.
So the first case is the empty dictionary, in msgpack this is serialized as:
>>> msgpack.packb({})
'\x80'
That's less than 4 bytes, and all BSONs are at least 4 bytes, so that can't be ambiguous. Now let's look at a dictionary with some content. Another thing we know about our payloads is that all the keys in the dictionaries are strings, and that the keys are alphanumeric or underscores. Looking at the msgpack spec, the smallest key (interpreted as its serialized integer value) that could exist is "0", since "0" has the lowest ASCII value of any letter, number, or underscore. Further, from the msgpack spec we know that the number 0 serializes as a single byte, so that will be the key's value. Let's see where this gets us:
>>> msgpack.packb({"0": 0})
'\x81\xa10\x00'
A 4 byte result, perfect, this is the smallest prefix we can generate, let's see how many bytes this would be:
>>> struct.unpack('<l', '\x81\xa10\x00')
(3187073,)
3187073 bytes, or a little over 3 MB. To be honest I'm not sure we have a key that starts with a number, let's try with the key "a":
>>> msgpack.packb({"a": 0})
'\x81\xa1a\x00'
>>> struct.unpack('<l', '\x81\xa1a\x00')
(6398337,)
A little over 6 MB. Since I know that none of the payloads we store are anywhere close to this large, we can safely store either serialization format, and be able to interpret the result unambiguously as one or the other.
So our final detection code looks like:
def deserialize(s):
if len(s) >= 4 and struct.unpack('<l', s[:4])[0] == len(s):
return BSON(s).decode()
else:
return msgpack.unpackb(s)
If this sounds like a fun kind of the thing to do, you should apply to come work with me at Rdio.
You can find the rest here. There are view comments.
I'm trying a new format for content, instead of writing I've recorded some audio, I think this will be a better format for my to share ideas. You can find the audio here.
PS: If you know a better workflow for sharing audio like this, please let me know.
You can find the rest here. There are view comments.
At 10:40 AM this morning Linux on the desktop was pronounced dead by Dr. Randall Gnu at GPL Memorial Hospital in Portland, Oregon. The cause of death is, as yet, unknown but it is believed to be the result of two decades of constant exposure to FUD.
Friends and colleagues were shocked and saddened at the loss of Linux on the desktop. Said one, "It was so shocking to hear it passed away, it had been doing so well lately, more users than ever, more non-technical users". Said another, "People always say, 'I never expected to get this news', but I really never expected it."
Linux on the desktop had a rocky two decade history, however the last five years had been characterized by unprecedented growth.
The family has requested that in lieu of flowers donations be made to the Linux Foundation.
You can find the rest here. There are view comments.
This is a response to http://pwang.wordpress.com/2012/07/11/does-the-compiler-know-best/ if you haven't read it yet, start there.
For lack of any other way to say it, I disagree with nearly every premise presented and conclusion derived in Peter's blog post. The post itself doesn't appear to have any coherent theme, besides that PyPy is not the future of Python, so I'll attempt to reply to Peter's statements more or less in order.
First, and perhaps most erroneously, he claims that "PyPy is an even more drastic change to the Python language than Python3". This is wrong. Complete and utterly. PyPy is in fact not a change to Python at all, PyPy faithfully implements the Python language as described by the Python language reference, and as verified by the test suite. Moreover, this is a statement that would apply equally to Jython and IronPython. It is pure, unadulterated FUD. Peter is trying to extend the definition of the Python language to things that it simple doesn't cover, such as the C-API and what he thinks the interpreter should look like (to be discussed more).
Second, he writes, "What is the core precept of PyPy? It’s that “the compiler knows best”." This too, is wrong. First, PyPy's central thesis is, "any task repeatedly performed manually will be done incorrectly", this is why we have things like automatic insertion of the garbage collector, in preference to CPython's "reference counting everywhere", and automatically generating the just in time compiler from the interpreter, in preference to Unladen Swallow's (and almost every other language's) manual construction of it. Second, the PyPy developers would never argue that the compiler knows best, as I alluded to in this post's title. That doesn't mean you should quit trying to write intelligent compilers, 1) the compiler often knows better than the user, just like with C, while it's possible to write better x86 assembler than GCC for specific functions, over the course of a large project GCC will always win, 2) they aren't mutually exclusive, having an intelligent compiler does not prohibit giving the user more control, in fact it's a necessity! There are no pure-python hints that you can give to CPython to improve performance, but these can easily be added with PyPy's JIT.
He follows this by saying that in contrast to PyPy's (nonexistent) principle of "compiler knows best" CPython's strength is that it can communicate with other platforms because its inner workings are simple. These three things have nothing to do with each other. CPython's interoperability with other platforms is a function of it's C-API. You can build an API like this on top of something monstrously complicated too, look at JNI for the JVM. (I don't accept that PyPy is so complex, but that's another post for another time.) In any event, the PyPy developers are deeply committed to interoperability with other platforms, which is why Armin and Maciej have been working on cffi: http://cffi.readthedocs.org/en/latest/index.html
The next paragraph is one of the most bizarre things I've ever read. He suggests that if you do want the free performance gains PyPy promises you should just build a a Python to JS compiler and use Node.js. I have to assume this paragraph is a joke not meant for publication, because it's nonsense. First, I've been told by the scientific Python community (of which Peter is a member) that any solution that isn't backwards compatible with a mountain of older platforms will never be adopted. So naturally his proposed solution is to throw away all existing work. Next, he implies that Google, Mozilla, Apple, and Microsoft are all collaborating on a single Javascript runtime which is untrue, in fact they each have their own VM. And V8, the one runtime specifically alluded to via Node.js, is not, as he writes, designed to be concurrent; Evan Phoenix, lead developer of Rubinius, comments, "It's probably the least concurrent runtime I've seen."
He then moves on to discussing the transparency of the levels involved in a runtime. Here I think he's 100% correct. Being able to understand how a VM is operating, what it's doing, what it's optimizing, how it's executing is enormously important. That's why I'm confused that he's positioning this as an argument against PyPy, as we've made transparency of our system incredibly important. We have the jitviewer, a tool which exposes the exact internal operations and machine code generated for everything PyPy compiles, which can be correlated to a individual line of Python code. We also have a set of hooks into the JIT to be able to programatically inspect what's happening, including writing your own, pure Python, optimization passes: http://pypy.readthedocs.org/en/latest/jit-hooks.html!
That's all I have. Hope you enjoyed.
You can find the rest here. There are view comments.
First, I have to apologize, because this post is incredibly self-serving, I believe everything I write here applies to myself, and if you do what I suggest, it will be to my benefit. I believe every single word I'm writing, but you have no way of knowing that, except trusting me. I can only hope I've written persuasively enough.
If you've ever asked an open source developer why they haven't fixed a bug or added a feature, the answer probably fell into one of three categories:
(Number may not add up to 100% due to rounding).
But why do you care? If you've made your way to my blog, it's incredibly likely you're a direct user of open source software. First, I think it'd be nice if you gave back. You're under no obligation to legally, morally, or otherwise, but I think it'd be a nice thing to do.
Second, and far more importantly: open source is an obscenely efficient way to develop software. I'm pretty sure if you compared the total person hours invested in V8, SpiderMonkey, the Hotspot JVM, and PyPy you'd find that PyPy had fewer person hours than any of those, by an order of magnitude, at least. And yet we can discuss them in the same sentence.
So here's the idea, there are a lot of developers (Github claims 1.7 million users). If we each spent a little of our money to pay a few open source developer's salaries, let's see what they could build if they did have the time.
If that sounds at all interesting to you, you should go over to Gittip, and help fund someone. I'm not sure Gittip is the perfect implementation of this idea, or even a good one. But I'm glad someone's trying it, because I want to see it work.
You can find the rest here. There are view comments.
5 years, 2 months, and 28 days ago I dropped out of high school, and in a few hours I'm going to graduate from college. This journey has been surreal, and that it's coming to an end hasn't even begun to sink in. I lack the words to express my thanks to my family and friends who've been with me through this.
However, I think I owe an even greater thanks to the open source communities I've worked with. Family and friends you have a mutual obligation to. But open source communities are all volunteer efforts, and I've often voiced that users of open source have no obligations to give back, that the freedom we offer is the freedom to give nothing back. And yet people do give back, again and again, in ways that completely floor me. Without these communities I wouldn't be anywhere near where I am today, they've given me friends, professional success, the opportunity to travel the world and do something I love, and most importantly, the sense that the work I've done has helped people.
For several months I've known that I wanted to write a "graduation post", but I struggled to come up with any particular narrative for it. I still don't have one, so instead of writing about anything in particular, I'm just going to offer some things I've learned:
Never be threatened: When people are threatened or scared, it affects their decision making. My solution to this has been to try to make myself unthreatenable. For example, I got through school by knowing that I was good enough at what I did that I could get a job without it, I avoid being threatened financially by saving effectively. I've found that this is more or less a necessary condition to being happy.
Be willing to walk away: This is probably an extension of the first one, but if you're anything like me you make mistakes, a lot of them, with varying degrees of badness. The only way not to get bogged down by them is to be willing to walk away from them. This is not to advocate abandoning something or someone, but rather that there is no hole too deep, no sunk cost too great, to not be worth leaving.
Help others to be lucky: I've been extraordinarily lucky. I define luck as the result of a decision made without the ability to predict it's consequences, given I dropped out of high school I was lucky to be able to get into college, even though I made a lot of good decisions between those two events (to this day I assume my acceptance at college was the result of a clerical error). The 9th season of Scrubs is not well liked by Scrubs fans, but it contains one of my favorite bits of dialog: "I've had, like, 15 second chances in my life. You deserve at least one, right?" The best I can do from here is to help someone else to be as lucky to get a second shot as I was.
The people are all that matter: I remember after a school trip a friend once told me that that he'd really loved where we'd been, and that he couldn't wait to go back. And I remember telling him, "No, this place was boring as hell, but we went with our entire class, all our really good friends, of course it was awesome". Just about everything worth doing is defined by the people you do it with, pick good ones.
If you can put a number to it, it probably doesn't count for much: I used to love numbers: IQs, GPAs, salaries, anything that you could represent as a number and compare I thought was awesome. But more and more I've found that they measure less and less. I can't assign a number to what makes someone a good programmer or a good friend. But as a society we're obsessed with this, because it gives us an illusion of accountability, but really it just makes us arbitrary. Because I'm pretty sure if you listed the traits that make someone a great programmer, you'd be picking a lot of things from two opposite sides of a spectrum, and a bunch from the middle as well.
We're all people: And in my experience most people are at least a little different from each. Slavish defferal to these, or any other pieces of advice probably won't work for you, nor will treating a large group of people as anything other than many different individuals.
It's been more than 5 years since I made the biggest decision of my life, and I've been fortunate enough to have enough opportunities for a life time in this short time. All I can hope for for my future is that absolutely nothing changes: that I still get paid to do what I love, that I stay in contact with good friends, and that I'm able to pay it forward.
You can find the rest here. There are view comments.
Polyglot programming (the practice of knowing and using many programming languages) seems to be all the rage these days. Its adherents claim two benefits:
I'm not going to dispute either of these. Well, maybe the second I'll argue with a little: I think you can get most of the benefits by using different paradigms within the same multi-paradigm language, and I'm a bit skeptical of the global benefits (unless you're the type of person who likes writing FORTRAN in Javascript). But I digress, like I said, I think those are both fair claims.
What I don't like is the conclusion that this means you should always use the right tool for the job. What, you the astute reader asks, does this mean you think we should use the wrong tool for the job? No, that would be idiotic, it means I think sometimes using the less optimal tool for the job carries overall benefits.
So what are the dangers of being a polyglot programmers (or the benefits of not being one, if you will)?
Using multiple languages (or any technology) stresses your operations people. It's another piece they have to maintain. If you've got a nice JVM stack, top to bottom, with nice logging and monitoring do you think your ops people really want to hear that they need to duplicate that setup so you can run three Ruby cron jobs? No, they're going to tell you to suck it up and write it up and either see if JRuby works or use Clojure or something, because 1% of your company's code isn't worth doubling their work.
Another risk is that it raises the requirements for all the other developers on the project. Martin Golding said, "Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live." Imagine when you leave your job and the next guy finds out you decided to write some data analysis scripts in APL (for those of you who don't remember, APL is that lovely language that doesn't use ASCII characters). It's fine to use APL if that's something you can require of new hires, it's not fine when your job says "Python developer" (it may actually work for Perl developers, but I assure you it'll be purely coincidental). Learning a new language is hard, learning to write it effectively is harder. Learning a new language for every script you have to maintain is downright painful, and once you know all of them, context switching isn't free for either humans or computers.
I'm not saying write everything in one language, that'd probably leave you writing a lot of code in very suboptimal languages. But choose two or three, not ten. Your ops people will thank you, and so will the guys who have to maintain your code in a decade. At DjangoCon this year Glyph Lefkowitz actually went farther, he argued that not just the code you write, but your entire technology stack should be in one language. But that's a separate discussion, you should watch the video though.
Also, because I'm a big fan of The West Wing, I'd be remiss if I used the word polyglot this many times without linking to a great scene.
You can find the rest here. There are view comments.
Raymond Hettinger recently asked on twitter what people thought del defaultdict()[k] did for a k that didn't exist in the dict. There are two ways of thinking about this, one is, "it's a defaultdict, there's always a value at a key, so it can never raise a KeyError", the other is, "that only applies to reading a value, this should still raise an error". I initially spent several minutes considering which made more sense, but I eventually came around to the second view, I'm going to explain why.
The Zen of Python says, "Errors should never pass silently." Any Java programmer who's seen NullPointerException knows the result of passing around invalid data, rather than propagating an error. There are two cases for trying to delete a key which doesn't exist in a defaultdict. One is: "this algorithm happens to sometimes produce keys that aren't there, not an issue, ignore it", the other is "my algorithm has a bug, it should always produce valid keys". If you don't raise a KeyError the first case has a single line of nice code, if you do raise an error they have a boring try/ except KeyError thing going on, but no big loss. However, if an error isn't raised and your algorithm should never produce nonexistent keys, you'll be silently missing a large bug in your algorithm, which you'll have to hope to catch later.
The inconvenience of ignoring the KeyError to the programmer with the algorithm that produces nonexistent keys is out weighed by the potential for hiding a nasty bug in the algorithm of the programmer who's code should never produce these. Ignoring an exception is easy, trying to find the bug in your algorithm can be a pain in the ass.
You can find the rest here. There are view comments.
I just delivered a talk at RCOS (the open source center on campus) on some of the work I've been doing this semester on PyPy's implementation of NumPy. You can find the slides here, they're built using html5slides, which is pretty swell (for a tool that makes me write HTML directly that is).
You can find the rest here. There are view comments.
At PyCodeConf I had a very interesting discussion with Nick Coghlan which helped me understand something that had long frustrated me with programming languages. Anyone who's ever taught a new programmer Java knows this, but perhaps hasn't understood it for what it is. This thing that I hadn't been appreciating was the distinction some programming languages make between the language that exists at compile time, and the language that exists at run-time.
Take a look at this piece of Java code:
class MyFirstProgram {
public static void main(String[] args) {
System.out.println("Hello World!");
}
}
Most people don't appreciate it, but you're really writing in two programming languages here, one of these languages has things like class and function declarations, and the other has executable statements (and yes, I realize Java has anonymous classes, they don't meaningfully provide anything I'm about to discuss).
Compare that with the (approximately) equivalent Python code:
def main():
print "Hello World"
if __name__ == "__main__":
main()
There's a very important thing to note here, we have executable statements at the top level, something that's simply impossible in Java, C, or C++. They make a distinction between the top level and your function's bodies. It follows from this that the function we've defined doesn't have special status by virtue of being at the top level, we could define a function or write a class in any scope. And this is important, because it gives us the ability to express things like decorators (both class and function).
Another example of this distinction that James Tauber pointed out to me is the import statement. In Python is it a line of executable code which invokes machinery in the VM to find a module and load it into the current namespace. In Java it is an indication to the compiler that a certain symbol is in scope, it's never executed.
Why would anyone care about this distinction though? It's clearly possibly to write programs in languages on both ends of the spectrum. It appears to me that the expressiveness of a programming language is really a description of what the distance between the compile time language and the runtime language is. Python stands on one end, with no distinction, whereas C/C++/Java stand on the other, with a grand canyon separating them.
But what about a language in the middle? Much of PyPy's code is written in a language named RPython. It has a fairly interesting property though, its run-time language is pretty close to Java in semantics, it's statically typed (though type inferenced), it's compile time language is Python. In practice this means you get many of the benefits in expressiveness as you do from using Python itself. For example you can write a decorator, or generate a class. A good example of this power is in PyPy's NumPy implementation. We're able to create the code for doing all the operations on different dtypes (NumPy's name for the different datatypes its arrays can represent) dynamically, without needing to resort to code generation or repeating ourselves, we're able to rely on Python as our compile time (or meta-programming) language. The "in-practice" result of this is that writing RPython feels much more like writing Python than it does like writing Java, even though most of your code is written under the same constraints (albeit without the need to explicitly write types).
The distinction between compile-time and run-time in programming languages results in both more pain for programmers, as well as more arcane structures to explain to new users. I believe new languages going forward should make it a goal to either minimize this difference (as Python does) or outfit languages with more powerful compile-time languages which give them the ability to express meta-programming constructs.
You can find the rest here. There are view comments.
Thinking about writing your own Python implementation? Congrats, there are plenty out there [1], but perhaps you have something new to bring to the table. Writing a fast Python is a pretty hard task, and there's a lot of stuff you need to keep in mind, but if you're interested in forging ahead, keep reading!
First, you'll need to write yourself an interpreter. A static compiler for Python doesn't have enough information to do the right things [2] [3], and a multi-stage JIT compiler is probably more trouble than it's worth [4]. It doesn't need to be super fast, but it should be within 2x of CPython or so, or you'll have lost too much ground to make up later. You'll probably need to write yourself a garbage collector as well, it should probably be a nice, generational collector [5].
Next you'll need implementations for all the builtins. Be careful here! You need to be every bit as good as CPython's algorithms if you want to stand a chance, this means things like list.sort() keeping up with Timsort [6], str.__contains__ keeping up with fast search [7], and dict.__getitem__ keeping up with the extremely carefully optimized Python dict [8].
Now you've got the core language, take a bow, most people don't make it nearly this far! However, there's still tons of work to go, for example you need the standard library if you want people to actually use this thing. A lot of the stdlib is in Python, so you can just copy that, but some stuff isn't, for that you'll need to reimplement it yourself (you can "cheat" on a lot of stuff and just write it in Python though, rather than C, or whatever language your interpreter is written in).
At this point you should have yourself a complete Python that's basically a drop-in replacement for CPython, but that's a bit slower. Now it's time for the real work to begin. You need to write a Just in Time compiler, and it needs to be a good one. You'll need a great optimizer that can simultaneously understand some of the high level semantics of Python, as well as the low level nitty gritty of your CPU [9].
If you've gotten this far, you deserve a round of applause, not many projects make it this far. But your Python probably still isn't going to be used by the world, you may execute Python code 10x faster, but the Python community is more demanding than that. If you want people to really use this thing you're going to have to make sure their C extensions run. Sure, CPython's C-API was never designed to be run on other platforms, but you can make it work, even if it's not super fast, it might be enough for some people [10].
Finally, remember that standard library you wrote earlier? Did you make sure to take your time to optimize it? You're probably going to need to take a step back and do that now, sure it's huge, and people use every nook and cranny of it, but if you want to be faster, you need it to be faster too. It won't do to have your bz2 module be slower, tarnishing your beautiful speed results [11].
Still with me? Congratulations, you're in a class of your own. You've got a blazing fast Python, a nicely optimized standard library, and you can run anyone's code, Python or C. If this ballad sounds a little familiar, that's because it is, it's the story of PyPy. If you think this was a fun journey, you can join in. There are ways for Python programmers at every level to help us, such as:
Hope to see you soon [12]!
| [1] | CPython, IronPython, Jython, PyPy, at least! |
| [2] | http://code.google.com/p/shedskin/ |
| [3] | http://cython.org/ |
| [4] | http://code.google.com/p/v8/ |
| [5] | http://docs.python.org/c-api/refcounting.html |
| [6] | http://hg.python.org/cpython/file/2.7/Objects/listsort.txt |
| [7] | http://effbot.org/zone/stringlib.htm |
| [8] | http://hg.python.org/cpython/file/2.7/Objects/dictnotes.txt |
| [9] | http://code.google.com/p/unladen-swallow/ |
| [10] | http://code.google.com/p/ironclad/ |
| [11] | https://bugs.pypy.org/issue733 |
| [12] | http://pypy.org/contact.html |
I gave a talk at DjangoCon Europe 2011 on using Django and PyPy (with another talk to be delivered tomorrow!), you can get the slides right on bitbucket, and for those who saw the talk, the PyPy compatibility wiki is here.
You can find the rest here. There are view comments.
For the past nearly two years I've been an independent contractor working primarily for Eldarion, and it's been fantastic, I can't say enough good things. However, this summer I'm shaking things up a bit and heading west. I'll be an intern at Quora this summer. They're a Python shop, and it's going to be my job to make everything zippy. Specifically I'll be making the site run on PyPy, and then tuning the hell out of both their codebase and PyPy itself. I'm super excited about the chance to get a major site running on PyPy, and to contribute as many performance improvements upstream as possible.
You can find the rest here. There are view comments.