metaprogramming and politics

Decentralize. Take the red pill.

Monkeypatching in unit tests, done right

with 15 comments

[updated, thanks to marius].

I am currently preparing my testing tutorials for Pycon and here is an example i’d lke to share already.

The problem: In a test function we want to patch an Environment variable and see if our application handles something related to it correctly. The direct approach for doing this in a test function might look like this:

def test_envreading(self):
    old = os.environ['ENV1']
    os.environ['ENV1'] = 'myval'
    try:
        val = myapp().readenv()
        assert val == "myval"
    finally:
        os.environ['ENV1'] = old

If we needed to do this several times for test functions we’d have a lot of repetetive boilerplatish code. The try-finally and undo-related code does not even take into account that ENV1 might not have been set originally.

Most experienced people would use setup/teardown methods to get less-repetetive testing code. We might end up with something slightly more general like this:

def setup_method(self, method):
    self._oldenv = os.environ.copy()

def teardown_method(self, method):
    os.environ.clear()
    os.environ.update(self._oldenv)

def test_envreading(self):
    os.environ['ENV1'] = "myval"
    val = myapp().readenv()
    assert val == "myval"

This avoids repetition of setup code but it scatters what belongs to the test function across three functions. All other functions in the Testcase class will get the service of a preserved environment although they might not need it. If i want to move away this testing function i will need to take care to copy the setup code as well. Or i start subclassing Test cases to share code. If we then start to need modifying other dicts or classes we have to add code in three places.

Monkeypatching the right way

Here is a version of the test function which uses pytest’s monkeypatch` plugin. The plugin does one thing: it provides a monkeypatch object for each test function that needs it. The resulting test function code then looks like this:

def test_envreading(self, monkeypatch):
    monkeypatch.setitem(os.environ, 'ENV1', 'myval')
    val = myapp().readenv()
    assert val == "myval"

Here monkeypatch.setitem() will memorize old settings and modify the environment. When the test function finishes the monkeypatch object restores the original setting. This test function is free to get moved across files. No other test function or code place is affected or required to change when it moves.

Let’s take a quick look at the “providing” side, i.e. the pytest_monkeypatch.py plugin which provides “Monkeypatch” instances to test functions. It makes use of pytest’s new pyfuncarg protocol.

The plugin itself is free to get refined and changed as well, without affecting the existing test code. The following 71 lines of code make up the plugin, including tests:

class MonkeypatchPlugin:
    """ setattr-monkeypatching with automatical reversal after test. """
    def pytest_pyfuncarg_monkeypatch(self, pyfuncitem):
        monkeypatch = MonkeyPatch()
        pyfuncitem.addfinalizer(monkeypatch.finalize)
        return monkeypatch

notset = object()

class MonkeyPatch:
    def __init__(self):
        self._setattr = []
        self._setitem = []

    def setattr(self, obj, name, value):
        self._setattr.insert(0, (obj, name, getattr(obj, name, notset)))
        setattr(obj, name, value)

    def setitem(self, dictionary, name, value):
        self._setitem.insert(0, (dictionary, name, dictionary.get(name, notset)))
        dictionary[name] = value

    def finalize(self):
        for obj, name, value in self._setattr:
            if value is not notset:
                setattr(obj, name, value)
            else:
                delattr(obj, name)
        for dictionary, name, value in self._setitem:
            if value is notset:
                del dictionary[name]
            else:
                dictionary[name] = value


def test_setattr():
    class A:
        x = 1
    monkeypatch = MonkeyPatch()
    monkeypatch.setattr(A, 'x', 2)
    assert A.x == 2
    monkeypatch.setattr(A, 'x', 3)
    assert A.x == 3
    monkeypatch.finalize()
    assert A.x == 1

    monkeypatch.setattr(A, 'y', 3)
    assert A.y == 3
    monkeypatch.finalize()
    assert not hasattr(A, 'y')


def test_setitem():
    d = {'x': 1}
    monkeypatch = MonkeyPatch()
    monkeypatch.setitem(d, 'x', 2)
    monkeypatch.setitem(d, 'y', 1700)
    assert d['x'] == 2
    assert d['y'] == 1700
    monkeypatch.setitem(d, 'x', 3)
    assert d['x'] == 3
    monkeypatch.finalize()
    assert d['x'] == 1
    assert 'y' not in d

def test_monkeypatch_plugin(testdir):
    sorter = testdir.inline_runsource("""
        pytest_plugins = 'pytest_monkeypatch',
        def test_method(monkeypatch):
            assert monkeypatch.__class__.__name__ == "MonkeyPatch"
    """)
    res = sorter.countoutcomes()
    assert tuple(res) == (1, 0, 0), res

I can also imagine some nice plugin which supports mock objects – patching methods with some preset behaviour or tracing calls between components.

have fun, holger

Written by holger krekel

March 3, 2009 at 1:48 pm

15 Responses

Subscribe to comments with RSS.

  1. A few comments. Only #5 is important, the rest are nitpicks. Well, #4 is also sort of important.

    1. You call monkeypatch.setitem in your very first example; I assume you meant to write os.environ['ENV1'] = ‘myval’ instead.

    2. An example using a with statement is conspicuously missing from this post, raising suspicions that you may be building a strawman.

    3. The setup/teardown example is not assigning a value to os.environ['ENV1']

    4. monkeypatch.setitem with automatic cleanup is ***awesome*** and negates most of point 2.

    5. if value is notset you should delattr(obj, name).

    6. Inserting new items in front of a list feels wrong (and inefficient, although that shouldn’t matter for the short lists used here); I’d suggest appending and then iterating in reversed() order.

    7. Why not use dicts instead of lists for _setattr and _setitem?

    Marius Gedminas

    March 3, 2009 at 6:17 pm

    • i fixed 1, 3, 5 [i was somehow using the wrong plugin version].

      2: i am not using the with statement because i work with many python versions and want py.test to work against 2.3 and 2.4 as well.

      6/7: yes i wanted to keep it simple and not think about how to hash obj/key.

      thanks for your fine feedback! holger

      hpk42

      March 3, 2009 at 6:37 pm

      • Supporting older Python versions is a legitimate concern. I just think it would be good to mention it explicitly (which you’ve now done in the comment) rather than silently ignore the with statement.

        I hadn’t considered that dict-like objects can support nonhashable objects as keys! Nice corner case.

        Marius Gedminas

        March 4, 2009 at 10:33 am

    • Another bit regarding the with statement: if i drop to pdb (with –pdb) i could stiill inspect the modified environment, but not when i use the with statement.

      However, fact is that py.test also finalizes too early – so monkeypatch.finalize() will have been called once PDB is reached. I anyway want to revisit the pdb-mechanism to make it work in cross-platform and distributed settings. this adds another reason.

      holger krekel

      March 5, 2009 at 8:32 pm

  2. this is cool. I do a similar thing in tests but I use a decorator. It looks like this:

    @with_patched_object(“smtplib”, “SMTP”, MockSMTP)
    def test_email():
    # pretend this is in app under test:
    import smtplib
    s = smtplib.SMTP()
    s.connect()
    # etc…

    It is also possible to implement this as a context manager so that when a user is on 2.5+ she can optionally use the with statement.

    def test_email():
    with patched_object(“smtplib”, “SMTP”, MockSMTP):
    s = smtplib.SMTP()
    # etc…

    (thus, the code could work in 2.4 with the caveat that you’d have to use it as a decorator)

    Kumar McMillan

    March 5, 2009 at 10:48 pm

    • sure, makes sense. A few notes:

      – if i want to patch an object that i create during the test a func decorator dosn’t help. So for Python >=2.5 the with statement is indeed needed. On the style side, this indents the code the more i need to patch things but that also limits how much people patch, so might be seen as a plus as well :)

      – there is a need to import the name “with_patched_object”. Part of my joy of using the “pyfuncarg” protocol is that i virtually have no imports for test-support code anymore.

      – (repeated from another comment) if i want to interactively debug on an error, the with statement will inadvertantly change back the patched object. might not be what i want in my pdb.

      But i see – i think i’d like to provide some more examples about the pyfuncarg protocol. Let’s also discuss it at Pycon at our nose/pytest session.
      cheers,
      holger

      holger krekel

      March 5, 2009 at 11:03 pm

    • Hey Kumar,

      My mock project has a decorator similar to yours (which is already implemented as a context manager as well).

      You can do:

      @patch(‘smtplib.SMTP’)
      def test_email(mock):
      ….

      It auto creates the mock for you (although it also allows you to pass in an explicit one if you want) and only requires one string instead of two to specify what you are mocking.

      MIchael Foord

      May 19, 2009 at 10:39 am

  3. You should test your code monkey patching and restoring static methods on classes. It may suffer from the same problem I recently blogged about.

    i.e. the following is broken for static methods:

    original = Class.method
    Class.method = original

    MIchael Foord

    May 19, 2009 at 10:36 am

    • Hi Michael. Yes, that is an issue. not sure what the best solution to it is, though. Maybe doing

      monkeypatch.setitem(cls.__dict__, name, staticmethod(myfunc))

      is good enough? It requires one to know that one is patching a static method. But trying to find this out automatically in “setattr” is probably too magical. Or do you have a good solution there?
      cheers,
      holger

      holger krekel

      May 19, 2009 at 5:55 pm

  4. You can just check the container and if the container is a class object then its property would be a staticmethod if its type is . Regular class functions would be or or ? … maybe I’m missing one. Could use some tests ;)

    Kumar McMillan

    May 19, 2009 at 6:29 pm

  5. [whoops, html got eaten. I meant:]

    You can just check the container and if the container is a class object then its attribute would be a staticmethod if the attribute type is . Regular class functions would be or or ? … maybe I’m missing one. Could use some tests ;)

    Kumar McMillan

    May 19, 2009 at 6:31 pm

  6. [sigh, no way to preview or delete comments, doh]

    You can just check the container and if the container is a class object then its attribute would be a staticmethod if the attribute type is type ‘function’. Regular class functions would be unbound method or type ‘property’ or ? … maybe I’m missing one. Could use some tests ;)

    Kumar McMillan

    May 19, 2009 at 6:33 pm

  7. In your non-Monkeypatch, the teardown_method() doesn’t clear os.environ, so if new keys have been defined during test execution they will persist. It should be defined like:

    def teardown_method(self, method):
    os.environ.update(self._oldenv)

    Stephen Ayotte

    October 14, 2014 at 5:06 pm

    • Well, I screwed that one up haha. Let’s try again:

      def teardown_method(self, method):
      os.environ.clear()
      os.environ.update(self._oldenv)

      Stephen Ayotte

      October 14, 2014 at 5:07 pm


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: