tox is a recent Python testing tool, by Holger Krekel. It's stated purpose is to make testing Python projects against multiple versions of Python (or different interpreters, like PyPy and Jython) much easier. However, it can be used for so much more. Yesterday I set it up for django-taggit, and it's an absolute dream, it automates testing against four different versions of Python, two different versions of Django, and it automates building the docs and checking for any warnings from Sphinx. I'll try to give a run through on what exactly you need to do to set this up with your project.
First create a tox.ini at the root of your project (i.e. in the same directory as your setup.py). Next create a [tox] section, and list out the enviroments you'd like to be tested (i.e. which Pythons):
[tox]
envlist =
py25, py26 , py27, pypy
The enviroments we've listed out are a few of the ones included with tox, they point at specific versions of Python and use the default testing setup. Now add a [testenv] section which will tell tox how to actually run your tests:
[testenv]
commands =
python setup.py test
deps =
django==1.2.3
commands is the list of commands tox will run, and deps specifies any dependencies that are needed to run the tests (tox creates a virtualenv for each enviroment and doesn't include system wide site-packages, so you need to make sure you list everything needed by default here). If you want to use this same python setup.py test formulation you'll need to be using setuptools or distribute for your setup.py and provide the test_suite argument, Eric Holscher provides a good run down for how to do this for Django projects.
Now you should be able to just type tox into your command line and it will try to run your tests in each of the enviroments you specified. Hopefully they're all passing (future test runs will go faster, for the first run it has to install all the dependencies). The next thing you may want to do is get it setup to build your documentation. To do this create a [testenv:docs] section:
[testenv:docs]
changedir = docs
deps =
sphinx
commands =
sphinx-build -W -b html -d {envtmpdir}/doctrees . {envtmpdir}/html
This tells tox a few things. First changedir tells it that to run these commands it should cd into the docs/ directory (if you're docs live elsewhere, change as appropriate). Next it has sphinx as a dependency. Finally the commands invoke sphinx-build, -W makes warnings into errors (so you get an approrpiate failure status code), -b html uses the HTML builder, and the rest of the parameters tell Sphinx where the docs live and to put the output in the temporary directory that tox creates for each env.
Now all you need to do is add docs to the envlist, and a tox run will build your documentation.
The last thing you might want to do is set it up to test against multiple versions of a package (such as Django 1.1, Django 1.2, and trunk). To do this create another section whose name includes both the Python version and dependency version, e.g. [testenv:py25-trunk]. In it place:
[testenv:py25-trunk]
basepython = python2.5
deps =
http://www.djangoproject.com/download/1.3-alpha-1/tarball/
This "inherits" from the default testenv, so it still has its commands, but we specify the basepython indicating this testenv is for python 2.5, and a different set of dependencies, here we're using the Django 1.3 alpha. You'll need to do a bit of copy-paste and create one of these for each version of Python you're testing against, and make sure to add each of these to the envlist.
At this point you should have a lean, mean, testing setup. With one command you can test your package with different dependencies, different pythons, and build your documentation. The tox documentation features tons of examples so you should use it as a reference.
You can find the rest here. There are view comments.
Testing is important, I've been a big advocate of writing tests for a while, however when you've got tests you need to run them. This is a big problem in open source, Django works on something like six versions of Python (2.4, 2.5, 2.6, 2.7, Jython, and PyPy), 4 databases (SQLite, PostgreSQL, MySQL, Oracle, plus the GIS backends, and external backends), and I don't even know how many operating systems (at least the various Linuxes, OS X, and Windows). If I tried to run the tests in all those configurations for every commit I'd go crazy. Reusable applications have it even worse, ideally they should be tested under all those configurations, with each version of Django they support. For a Django application that wants to work on Django 1.1, 1.2, and all of those interpreters, databases, and operating systems you've got over 100 configurations. Crazy. John Resig faced a similar problem with jQuery (5 major browsers, multiple versions, mobile and desktop, different OSs), and the result was Test Swarm (note that at this time it doesn't appear to be up), an automated way for people to volunteer their machines to run tests. We need something like that for Python.
It'd be nice if it were as simple as users pointing their browser at a URL, but that's not practical with Python: the environments we want to test in are more complex than what can be detected, we need to know what external services are available (databases, for example). My suggestion is users should maintain a config file (.ini perhaps) somewhere on their system, it would say what versions of Python are available, and what external services are available (and how they can be accessed, e.g. DB passwords). Then the user downloads a bootstrap script and runs it. This script sees what services the user has available on their machine and queries a central server to see what tests need to be run, given the configuration they have available. The script downloads a test, creates a virtualenv, and does whatever setup it needs to do (e.g. writing a Django settings.py file given the available DB configuration), and runs the tests. Finally it sends the test results back to the central server.
It's very much like a standard buildbot system, except any user can download the script and start running tests. There are a number of problems to be solved, how do you verify that a project's tests aren't malicious (only allow trusted tests to start), how do you verify that the test results are valid, how do you actually write the configuration for a test suite? However, if solved I think this could be an invaluable resource for the Python community. Have a reusable app you want tested? Sign it up for PonySwarm, add a post-commit hook, and users will automatically run the tests for it.
You can find the rest here. There are view comments.
Lately I've had the opportunity to do some test-driven development with Django, which a) is awesome, I love testing, and b) means I've been working up a box full of testing utilities, and I figured I'd share them.
If you've done testing of views with Django you probably have some tests that look like:
def test_my_view(self):
response = self.client.get(reverse("my_url", kwargs={"pk": 1}))
response = self.client.post(reverse("my_url", kwargs={"pk": 1}), {
"key": "value",
})
This was a tad too verbose for my tastes so I wrote:
def get(self, url_name, *args, **kwargs):
return self.client.get(reverse(url_name, args=args, kwargs=kwargs))
def post(self, url_name, *args, **kwargs):
data = kwargs.pop("data", None)
return self.client.post(reverse(url_name, args=args, kwargs=kwargs), data)
Which are used:
def test_my_view(self):
response = self.get("my_url", pk=1)
response = self.post("my_url", pk=1, data={
"key": "value",
})
Much nicer.
The next big issue I had was logging in and out of multiple users was too verbose. I often want to switch between users, either to check different permissions or to test some inter-user workflow. That was solved with a simple context manager:
class login(object):
def __init__(self, testcase, user, password):
self.testcase = testcase
success = testcase.client.login(username=user, password=password)
self.testcase.assertTrue(
success,
"login with username=%r, password=%r failed" % (user, password)
)
def __enter__(self):
pass
def __exit__(self, *args):
self.testcase.client.logout()
def login(self, user, password):
return login(self, user, password)
This is used:
def test_my_view(self):
with self.login("username", "password"):
response = self.get("my_url", pk=1)
Again, a lot better.
Not quite a testing utility, but my app django-fixture-generator has made testing a lot easier for me. Fixtures are useful in getting data to work wit, but maintaining them is often a pain, you've got random scripts to generate them, or you just checkin some JSON to your repository with no way to regenerate it sanely (say if you add a new field to your model). django-fixture-generator gives you a clean way to manage the code for generating fixtures.
In general I've found context managers are a pretty awesome tool for writing clean, readable, succinct tests. I'm sure I'll have more utilities as I write more tests, hopefully someone finds these useful.
While doing some work today I realized that generating fixtures in Django is way too much of a pain in the ass, and I suspect it's a pain in the ass for a lot of other people as well. I also came up with an API I'd kind of like to see for it, unfortunately I don't really have the time to write the whole thing, however I'm hoping someone else does.
The key problem with writing fixtures is that you want to have a clean enviroment to generate them, and you need to be able to edit them in the future. In addition, I'd personally prefer to have my fixture generation specifically be imperative. I have an API I think I think solves all of these concerns.
Essentially, in every application you can have a fixture_gen.py file, which contains a bunch of functions that can generate fixtures:
from fixture_generator import fixture_generator
from my_app.models import Model1, Model2
@fixture_generator(Model1, Model2, requires=["my_app.other_dataset"])
def some_dataset():
# Some objects get created here
@fixture_generator(Model1)
def other_dataset():
# Some objects get created here
Basically you have a bunch of functions, each of which is responsible for creating some objects that will become a fixture. You then decorate them with a decorator that specifies what models need to be included in the fixture that results from them, and finally you can optionally specify dependencies (these are necessary because a dependency could use models which your fixture doesn't).
After you have these functions there's a management command which can be invoked to actually generate the fixtures:
$ ./manage.py generate_fixture my_app.some_dataset --format=json --indent=4
Which actually creates the clean database enviroment, handles the dependencies, calls the functions, and dumps the fixtures to stdout. Then you can redirect that stdout off to a file somewhere, for use in testing or whatever else people use fixtures for.
Hopefully someone else has this problem, and likes the API enough to build this. Failing that I'll try to make some time for it, but no promises when (aka if you want it you should probably build it).
You can find the rest here. There are view comments.