Your tests are not a benchmark

I get a lot of feedback about people’s experiences with PyPy. And a lot of it is really great stuff for example, “We used to leave the simulation running over night, now we take a coffee break”. We also get some less successful feedback, however quite a bit of that goes something like, “I ran our test suite under PyPy, not only was it not faster, it was slower!”. Unfortunately, for the time being, this is really expected, we’re working on improving it, but for now I’d like to explain why that is.

If you want to find out how fast PyPy (or any technology) is, sit down and write some benchmarks, I’ve got some advice on how to do that.