metaprogramming and politics

Decentralize. Take the red pill.

Archive for the ‘testing’ Category

Monkeypatching in unit tests, done right

with 16 comments

[updated, thanks to marius].

I am currently preparing my testing tutorials for Pycon and here is an example i’d lke to share already.

The problem: In a test function we want to patch an Environment variable and see if our application handles something related to it correctly. The direct approach for doing this in a test function might look like this:

def test_envreading(self):
    old = os.environ['ENV1']
    os.environ['ENV1'] = 'myval'
    try:
        val = myapp().readenv()
        assert val == "myval"
    finally:
        os.environ['ENV1'] = old

If we needed to do this several times for test functions we’d have a lot of repetetive boilerplatish code. The try-finally and undo-related code does not even take into account that ENV1 might not have been set originally.

Most experienced people would use setup/teardown methods to get less-repetetive testing code. We might end up with something slightly more general like this:

def setup_method(self, method):
    self._oldenv = os.environ.copy()

def teardown_method(self, method):
    os.environ.clear()
    os.environ.update(self._oldenv)

def test_envreading(self):
    os.environ['ENV1'] = "myval"
    val = myapp().readenv()
    assert val == "myval"

This avoids repetition of setup code but it scatters what belongs to the test function across three functions. All other functions in the Testcase class will get the service of a preserved environment although they might not need it. If i want to move away this testing function i will need to take care to copy the setup code as well. Or i start subclassing Test cases to share code. If we then start to need modifying other dicts or classes we have to add code in three places.

Monkeypatching the right way

Here is a version of the test function which uses pytest’s monkeypatch` plugin. The plugin does one thing: it provides a monkeypatch object for each test function that needs it. The resulting test function code then looks like this:

def test_envreading(self, monkeypatch):
    monkeypatch.setitem(os.environ, 'ENV1', 'myval')
    val = myapp().readenv()
    assert val == "myval"

Here monkeypatch.setitem() will memorize old settings and modify the environment. When the test function finishes the monkeypatch object restores the original setting. This test function is free to get moved across files. No other test function or code place is affected or required to change when it moves.

Let’s take a quick look at the “providing” side, i.e. the pytest_monkeypatch.py plugin which provides “Monkeypatch” instances to test functions. It makes use of pytest’s new pyfuncarg protocol.

The plugin itself is free to get refined and changed as well, without affecting the existing test code. The following 71 lines of code make up the plugin, including tests:

class MonkeypatchPlugin:
    """ setattr-monkeypatching with automatical reversal after test. """
    def pytest_pyfuncarg_monkeypatch(self, pyfuncitem):
        monkeypatch = MonkeyPatch()
        pyfuncitem.addfinalizer(monkeypatch.finalize)
        return monkeypatch

notset = object()

class MonkeyPatch:
    def __init__(self):
        self._setattr = []
        self._setitem = []

    def setattr(self, obj, name, value):
        self._setattr.insert(0, (obj, name, getattr(obj, name, notset)))
        setattr(obj, name, value)

    def setitem(self, dictionary, name, value):
        self._setitem.insert(0, (dictionary, name, dictionary.get(name, notset)))
        dictionary[name] = value

    def finalize(self):
        for obj, name, value in self._setattr:
            if value is not notset:
                setattr(obj, name, value)
            else:
                delattr(obj, name)
        for dictionary, name, value in self._setitem:
            if value is notset:
                del dictionary[name]
            else:
                dictionary[name] = value


def test_setattr():
    class A:
        x = 1
    monkeypatch = MonkeyPatch()
    monkeypatch.setattr(A, 'x', 2)
    assert A.x == 2
    monkeypatch.setattr(A, 'x', 3)
    assert A.x == 3
    monkeypatch.finalize()
    assert A.x == 1

    monkeypatch.setattr(A, 'y', 3)
    assert A.y == 3
    monkeypatch.finalize()
    assert not hasattr(A, 'y')


def test_setitem():
    d = {'x': 1}
    monkeypatch = MonkeyPatch()
    monkeypatch.setitem(d, 'x', 2)
    monkeypatch.setitem(d, 'y', 1700)
    assert d['x'] == 2
    assert d['y'] == 1700
    monkeypatch.setitem(d, 'x', 3)
    assert d['x'] == 3
    monkeypatch.finalize()
    assert d['x'] == 1
    assert 'y' not in d

def test_monkeypatch_plugin(testdir):
    sorter = testdir.inline_runsource("""
        pytest_plugins = 'pytest_monkeypatch',
        def test_method(monkeypatch):
            assert monkeypatch.__class__.__name__ == "MonkeyPatch"
    """)
    res = sorter.countoutcomes()
    assert tuple(res) == (1, 0, 0), res

I can also imagine some nice plugin which supports mock objects – patching methods with some preset behaviour or tracing calls between components.

have fun, holger

Advertisements

Written by holger krekel

March 3, 2009 at 1:48 pm

New Plugin architecture and plugins for py.test

with one comment

I just merged the plugin branch and am very happy about it. Part of the effort was driven by moving core functionality to become a plugin: Terminal reporting is now fully a plugin, contained in a single file including tests. It does it work solely by looking at testing events. Plugins can also add new aspects to tests files – for example the pytest_restdoc.py plugin adds ReST syntax, referential integrity and URL checking for Text files. (I used it for checking my blog post and its links, btw).

Pytest’s good old conftest.py files are still useful: you can define project or directory specific settings, including which plugins to use. For now, many old extensions should work unmodified, as exemplified by PyPy‘s extensive conftest.py files. It’s easy to port a conftest file to a plugin. In fact, you can first define a local "ConftestPlugin" and later move it to become a cross-project one – a matter of renaming the file and the class, done!

To serve as guiding examples, I drafted some initial plugins and implemented neccessary hooks within py.test core.

If you wan’t to get a feel on how plugins are implemented, here is the pytest_eventlog.py plugin which adds a command line option to allow logging of all testing events. It’s instructive to look at how it’s done as well as the output because it shows which testing events are generated.

class EventlogPlugin:
    """ log pytest events to a file. """

    def pytest_addoption(self, parser):
        parser.addoption("--eventlog", dest="eventlog",
            help="write all pytest events to the given file.")

    def pytest_configure(self, config):
        eventlog = config.getvalue('eventlog')
        if eventlog:
            self.eventlogfile = open(eventlog).open('w')

    def pytest_unconfigure(self, config):
        if hasattr(self, 'eventlogfile'):
            self.eventlogfile.close()
            del self.eventlogfile

    def pyevent(self, eventname, *args, **kwargs):
        if hasattr(self, 'eventlogfile'):
            print >>self.eventlogfile, eventname, args, kwargs
            self.eventlogfile.flush()

This plugin code is complete, except that the original pytest_eventlog.py file contains tests. The eventlog plugin methods above are called in the following way:

  • def pytest_addoption(self, parser) is called before
    commandline arguments are parsed.
  • def pytest_configure(self, config) is called after parsing
    arguments and before any reporting, collection or running
    of tests takes place.
  • def pytest_event(self, eventname, *args, **kwargs) is called
    for each testing event. Events have names and come with
    arguments which are supplied by the event producing site.
  • def pytest_unconfigure(self, config) is called after
    all test items have been processed.

If you want to start writing your own plugin, please use an svn checkout of:

http://codespeak.net/svn/py/trunk/

and activate it by e.g. python setup.py develop.

If you want to write a plugin named pytest_XYZ, you can tell pytest to use it by setting the environment variable PYTEST_PLUGINS=XYZ or by putting pytest_plugins = 'xyz' into a test module or conftest.py file.

A good way to contribute is to copy an existing plugin file to your home dir and put it somewhere into your PYTHONPATH. py.test will use your version instead of the default one and you can play with it untill you are happy (and see to also add some tests showing the new behaviour).

If you have questions or problems, you are invited to post here or to the py-dev mailing list. I’d definitely like to pluginize more of pytest and add hooks as needed and am happy for feedback and suggestions before i freeze the API for 1.0.

holger

Written by holger krekel

February 27, 2009 at 11:22 am

New way to organize Python test code

with 5 comments

py.test just grew a new way to provide test state for a test function. First the problem: those of us dealing with writing tests in the hundreds or thousands usually setup test state at class, method or module level. Then we access it indirectly, through self, local or global helper functions. For larger applications, this usually leads to scattered, complex and boilerplatisch test code. This then stands in the way of refactoring the original code base … but wait, weren’t tests meant to ease refactoring, not hinder it?

Here is the idea: Python Test functions use their function definition to state their needs and the test tools calls a function that provides the value. For example, consider this test function:


   def test_ospath(self, tempdir):
      # needs tempdir to create files etc.

.

py.test provides the value for tempdir by calling a matching method that looks like this:


  def pytest_pyfuncarg_tempdir(pyfuncitem):
      # use pyfuncitem to access test context, cmdline opts etc.

.

This matching provider function returns a value for tempdir that is then supplied to the test function. For more complex purposes, the pyfuncitem argument provides full access to the test collection process including cmdline options, test options, project specific configuration. You can write down this provider method in the test
module, in configuration files or in a plugin.

Once i started using this new paradigm, i couldn’t resist and refactored pytest’s own tests to use the new method everywhere. Here are my findings so far:

  • self contained test functions: i don’t need to wade through unneccessary layers and indirection of test setup.
  • fewer imports: my test modules don’t need to import modules that are only needed for setting up application state.
  • easy test state setup: I can place test support code in one place and i can grep for pytest_pyfuncarg_NAME. I can reuse this setup code easily across modules, directories or even projects. Think about providing test database object or mocking objects.
  • more flexible test grouping: I can logically group tests however i like, independently from test setup requirements. I found it very easy to shuffle test functions between classes or modules because they are rather self-contained.

Written by holger krekel

February 22, 2009 at 5:23 pm