metaprogramming and politics

Decentralize. Take the red pill.

Archive for the ‘testing’ Category

Running tests against multiple devices/resources (in parallel)

with one comment

devicesHow to best distribute tests against multiple devices or resources with pytest? This interesting question came up during my training in Lviv (Ukraine) at an embedded systems company. Distributing tests to processes can serve two purposes:

  • running the full test suite against each device to verify they all work according to the test specification
  • distributing the test load to several devices of the same type in order to minimize overall test execution time.

The solution to both problems is easy if you use two pytest facilities:

  • the general fixture mechanism: we write a fixture function which provides a device object which is pre-configured for use in tests.
  • the pytest-xdist plugin: we use it to run subprocesses and communicate configuration data for the device fixture from the master process to the subprocesses.

To begin with, let’s configure three devices that are each reachable by a separate IP address. We create a list of ip addresses in a file:

# content of devices.json
["192.168.0.1", "192.168.0.2", "192.168.0.3"]

We now create a local pytest plugin which reads the configuration data, implements a per-process device fixture and the master-to-slave communication to configure each subprocess according to our device list:

# content of conftest.py

import pytest

def read_device_list():
    import json
    with open("devices.json") as f:
        return json.load(f)

def pytest_configure(config):
     # read device list if we are on the master
     if not hasattr(config, "slaveinput"):
        config.iplist = read_device_list()

def pytest_configure_node(node):
    # the master for each node fills slaveinput dictionary
    # which pytest-xdist will transfer to the subprocess
    node.slaveinput["ipadr"] = node.config.iplist.pop()

@pytest.fixture(scope="session")
def device(request):
    slaveinput = getattr(request.config, "slaveinput", None)
    if slaveinput is None: # single-process execution
        ipadr = read_device_list()[0]
    else: # running in a subprocess here
        ipadr = slaveinput["ipadr"]
    return Device(ipadr)

class Device:
    def __init__(self, ipadr):
        self.ipadr = ipadr

    def __repr__(self):
        return "<Device ip=%s>" % (self.ipadr)

We can now write tests that simply make use of the device fixture by using its name as an argument to a test function:

# content of test_device.py
import time

def test_device1(device):
    time.sleep(2)  # simulate long test time
    assert 0, device

def test_device2(device):
    time.sleep(2)  # simulate long test time
    assert 0, device

def test_device3(device):
    time.sleep(2)  # simulate long test time
    assert 0, device

Let’s first run the tests in a single-process, only using a single device (also using some reporting option to shorten output):

$ py.test test_device.py -q --tb=line
FFF
================================= FAILURES =================================
/tmp/doc-exec-9/test_device.py:5: AssertionError: <Device ip=192.168.0.1>
/tmp/doc-exec-9/test_device.py:9: AssertionError: <Device ip=192.168.0.1>
/tmp/doc-exec-9/test_device.py:13: AssertionError: <Device ip=192.168.0.1>
3 failed in 6.02 seconds

As to be expected, we get six seconds execution time (3 tests times 2 seconds each).

Now let’s run the same tests in three subprocesses, each using a different device:

$ py.test --tx 3*popen --dist=each test_device.py -q --tb=line
gw0 I / gw1 I / gw2 I
gw0 [3] / gw1 [3] / gw2 [3]

scheduling tests via EachScheduling
FFFFFFFFF
================================= FAILURES =================================
E   AssertionError: <Device ip=192.168.0.1>
E   AssertionError: <Device ip=192.168.0.3>
E   AssertionError: <Device ip=192.168.0.2>
E   AssertionError: <Device ip=192.168.0.1>
E   AssertionError: <Device ip=192.168.0.3>
E   AssertionError: <Device ip=192.168.0.2>
E   AssertionError: <Device ip=192.168.0.3>
E   AssertionError: <Device ip=192.168.0.1>
E   AssertionError: <Device ip=192.168.0.2>
9 failed in 6.52 seconds

We just created three subprocesses each running three tests. Instead of 18 seconds execution time (9 tests times 2 seconds per test) we roughly got 6 seconds, a 3-times speedup. Each subprocess ran in parallel three tests against “its” device.

Let’s also run with load-balancing, i.e. distributing the tests against three different devices so that each device executes one test:

$ py.test --tx 3*popen --dist=load test_device.py -q --tb=line
gw0 I / gw1 I / gw2 I
gw0 [3] / gw1 [3] / gw2 [3]

scheduling tests via LoadScheduling
FFF
================================= FAILURES =================================
E   AssertionError: <Device ip=192.168.0.3>
E   AssertionError: <Device ip=192.168.0.2>
E   AssertionError: <Device ip=192.168.0.1>
3 failed in 2.50 seconds

Here each test runs in a separate process against its device, overall more than halfing the test time compared to what it would take in a single-process (3*2=6 seconds). If we had many more tests than subproceses than load-scheduling would distribute tests in real-time to the process which has finished executing other tests.

Note that the tests themselves do not need to be aware of the distribution mode. All configuration and setup is contained in the conftest.py file.

To summarize the behaviour of the hooks and fixtures in conftest.py:

  • pytest_configure(config) is called both on the master and each subprocess. We can distinguish where we are by checking for presence of config.slaveinput.
  • pytest_configure_node(node) is called for each subprocess. We can fill the slaveinput dictionary which the subprocess slave can then read via its config.slaveinput dictionary.
  • the device fixture only is called when a test needs it. In distributed mode, tests are only collected and executed in a subprocess. In non-distributed mode, tests are run single-process. The Device class is just a stub — it will need to grow methods for actual device communication. The tests can then simply use those device methods.

I’d like to thank Anton and the participants of my three day testing training in Lviv (Ukraine) for bringing up this and many other interesting questions.

 

I am giving another such professional testing course 25-27th November at the Python Academy in Leipzig. There are still two seats available. Me and other trainers can also be booked for on-site/in-house trainings worldwide.

Written by holger krekel

November 12, 2013 at 7:43 am

metaprogramming in Python: What CPython, PyPy, Pyramid, pytest and politics have in common …

leave a comment »

Metaprogramming in Python too often revolves around metaclasses, which are just a narrow application of the “meta” idea and not a great one at that. Metaprogramming more generally deals with reasoning about program code, about taking a “meta” stance on it.  A metaprogram takes a program as input, often just partial programs like functions or classes. Here are a few applications of metaprogramming:

  • CPython is a metaprogram written in C. It takes Python program code as input and interprets it, so that it runs at a higher level than C.
  • PyPy is a metaprogramm written in Python. It takes RPython program code as input and generates a C-level metaprogram (the PyPy interpreter) which itself interprets Python programs and takes another meta stance by generating Assembler pieces for parts of the interpreation execution. If you like, PyPy is a metaprogram generating metaprograms whereas CPython and typical compilers like GCC are “just” a metaprogram.
  • Pyramid is a metaprogram that takes view, model definitions and http-handling code as input and executes them, thereby raising code on a higher level to implement the “Pyramid application” language.
  • pytest is a metaprogram written in Python, taking test, fixture and plugin functions as input and executing them in a certain manner, thereby implementing a testing language.
  • metaclasses: in Python they allow to intercept class creation and introspect methods and attributes, amending their behaviour. Because metaclass-code usually executes at import time, it often uses global state for implementing non-trivial meta aspects.

Apart from these concrete examples, language compilers, testing tools and web frameworks all have metaprogramming aspects. Creating big or small “higher” level or domain-specific languages within Python is as a typical example of metaprogramming. Python is actually a great language for metaprogramming although it could be better.

In future blog posts i plan to talk about some good metaprogramming practise, particularly:

  • keep the layers/levels separate by good naming and API design
  • define a concise “language” for the programs you take as input
  • avoid creating global state in your metaprograms (and elsewhere)
    which can easily happen with meta-classes executing at import time

Lastly, i see metaprogramming at work not only when coding in a computer language. Discussing the legal framing for executing programs on the internet is some kind of metaprogramming, especially if you consider licensing and laws as human-interpreted code which affects how programs can be written, constructed and executed. In reverse, web applications increasingly affect how we interact with each other other, thereby implementing rules formerly dealt with in the arena of politics. Therefore, metaprogramming and politics are fundamentally connected topics.

have metafun, i. e. take fun stuff as input to generate more of it :) holger

Written by holger krekel

November 22, 2012 at 3:04 pm

If i were to design a new programming language …

with 18 comments

I’d see to base syntax and semantics on Python3, but strip and rebase it:

  • no C: implement the interpreter in RPython, get a JIT for free and implementation bits from PyPy’s Python interpreter (parsing, IO, etc.)
  • no drags-you-down batteries: lean interpreter core and a standard battery distro which is tested against the last N interpreter versions + current
  • no yield: use greenlets to implement all of what yield provides and more
  • no underlying blocking on IO: base it all on event loop, yet provide synchronous programming model through greenlets
  • no c-level API nor ctypes: use cffi to interface with c-libraries
  • no global state: just support state bound to execution context/stack
  • no GIL: support free threading and Automatic Mutual Exclusion for dealing with shared state
  • no setup.py: have a thought-through story and tools from the start for packaging, installation, depending/interfacing between packages
  • no import, no sys.modules: provide an object with which you can access other packages’s objects and introspect/interact with one’s own package
  • no testing as an afterthought: everything needs to be easily testable, empowered assert statement and branch-coverage supported from the core.
  • no extensibility as an afterthought: support plugins and loose coupling through builtin 1:N calling mechanism (event notification on steroids)
  • no unsafe code: support IO/CPU/RAM sandboxing as a core feature
  • no NIH syndrome: provide a bridge to a virtualenv’ed Python interpreter allowing to leverage existing good crap

Anything else? Probably! Discussion needed? Certainly. Unrealistic? Depends on who would participate — almost all of the above has projects, PEPs and code showcasing viability.

Btw, did you know that when we started PyPy we initially did this under the heading of “Minimal Python”? Some of the above ideas above and their underlying motivations were already mentioned when I invited to the first PyPy sprint almost 10 years ago:

http://mail.python.org/pipermail/python-dev/2003-January/032427.html

I learned since then that Python has more complex innards than it seems but i still believe it could be both simpler and more powerful.

holger

Written by holger krekel

November 13, 2012 at 3:29 pm

pylib 1.0.0 released: the testing-with-python innovations continue

with 5 comments

Took a few betas but finally i uploaded a 1.0.0 py lib release, featuring the mature and powerful py.test tool and "execnet-style" elastic distributed programming. With the new release, there are many new advanced automated testing features – here is a quick summary:

  • funcargs – pythonic zero-boilerplate fixtures for Python test functions :
    • totally separates test code, test configuration and test setup
    • ideal for integration and functional tests
    • allows for flexible and natural test parametrization schemes
  • new plugin architecture, allowing easy-to-write project-specific and cross-project single-file plugins. The most notable new external plugin is oejskit which naturally enables running and reporting of javascript-unittests in real-life browsers.
  • many new features done in easy-to-improve default plugins, highlights:
    • xfail: mark tests as "expected to fail" and report separately.
    • pastebin: automatically send tracebacks to pocoo paste service
    • capture: flexibly capture stdout/stderr of subprocesses, per-test …
    • monkeypatch: safely monkeypatch modules/classes from within tests
    • unittest: run and integrate traditional unittest.py tests
    • figleaf: generate html coverage reports with the figleaf module
    • resultlog: generate buildbot-friendly reporting output
  • distributed testing and elastic distributed execution:
    • new unified "TX" URL scheme for specifying remote processes
    • new distribution modes "–dist=each" and "–dist=load"
    • new sync/async ways to handle 1:N communication
    • improved documentation

The py lib continues to offer most of the functionality used by the testing tool in independent namespaces.

Some non-test related code, notably greenlets/co-routines and api-generation now live as their own projects which simplifies the installation procedure because no C-Extensions are required anymore.

The whole package should work well with Linux, Win32 and OSX, on Python 2.3, 2.4, 2.5 and 2.6. (Expect Python3 compatibility soon!)

For more info, see the py.test and py lib documentation:

http://pytest.org

http://pylib.org

have fun, holger

Written by holger krekel

August 4, 2009 at 10:05 am

Parametrizing Python tests, generalized.

with 3 comments

Parametrizing test runs is a kind of a hot topic with Python test tools. py.test recently grew a new pytest_generate_tests hook to parametrize tests. I am going to introduce it by providing ports of Michael Foord‘s recent experiments with parametrizing unittest.py test cases and an example from Rob Collins testscenarios unittest extension. The gist of the new hook is that it allows to easily implement and combine these schemes. It builds on the general idea of allowing python test functions to receive function arguments ("funcargs") – and defining mechanisms on how to provide them.

The parametrizer example, ported

The idea of Michael Foord‘s Parametrizer example is to define multiple sets of parameters and have specified test functions receive those arguments. Here is a direct port of Michael’s example to use py.test’s new hook:

#./test_parametrize.py
import py

def pytest_generate_tests(metafunc):
    # called once per each test function
    for funcargs in metafunc.cls.params[metafunc.function.__name__]:
        # schedule a new test function run with applied **funcargs
        metafunc.addcall(funcargs=funcargs)

class TestClass:
    params = {
        'test_equals': [dict(a=1, b=2), dict(a=3, b=3), dict(a=5, b=4)],
        'test_zerodivision': [dict(a=1, b=0), dict(a=3, b=2)],
    }

    def test_equals(self, a, b):
        assert a == b

    def test_zerodivision(self, a, b):
        py.test.raises(ZeroDivisionError, "a/b")

py.test automatically discovers both the pytest_generate_tests hook and the two test functions. For each test function it calls the hook, passing it a metafunc object which provides meta information about the test function and allows to add new tests during collection. Let’s see what just collecting the tests produces:

$ py.test --collectonly test_parametrize.py

<Module 'test_parametrize.py'>
  <Class 'TestClass'>
    <Instance '()'>
      <FunctionCollector 'test_equals'>
        <Function 'test_equals[0]'>
        <Function 'test_equals[1]'>
        <Function 'test_equals[2]'>
      <FunctionCollector 'test_zerodivision'>
        <Function 'test_zerodivision[0]'>
        <Function 'test_zerodivision[1]'>

So we collected 5 actual runs of test functions. Let now run the test functions:

$ py.test test_parametrize.py

========================= test session starts =========================
python: platform linux2 -- Python 2.6.2
test object 1: test_parametrize.py

test_parametrize.py F.F.F

============================== FAILURES ===============================
________________ TestClass.test_equals.test_equals[0] _________________

self = <test_parametrize.TestClass instance at 0x994f8ac>, a = 1, b = 2

    def test_equals(self, a, b):
>       assert a == b
E       assert 1 == 2

test_parametrize.py:14: AssertionError
________________ TestClass.test_equals.test_equals[2] _________________

self = <test_parametrize.TestClass instance at 0x994f8ac>, a = 5, b = 4

    def test_equals(self, a, b):
>       assert a == b
E       assert 5 == 4

test_parametrize.py:14: AssertionError
__________ TestClass.test_zerodivision.test_zerodivision[1] ___________

self = <test_parametrize.TestClass instance at 0x994f8ac>, a = 3, b = 2

    def test_zerodivision(self, a, b):
>       py.test.raises(ZeroDivisionError, "a/b")
E       ExceptionFailure: 'DID NOT RAISE'

test_parametrize.py:17: ExceptionFailure
================= 3 failed, 2 passed in 0.13 seconds =================

You can easily see the failing tests and the parameters that the tests received. It also showcases py.test traceback reporting but that’s for another discussion.

The parametrizer example, decorated

So, you say, what about having a decorator specifying test parameters? Here is the same example, letting our hook implement a decorator scheme:

#./test_parametrize2.py

import py

def params(funcarglist):
    def wrapper(function):
        function.funcarglist = funcarglist
        return function
    return wrapper

def pytest_generate_tests(metafunc):
    for funcargs in getattr(metafunc.function, 'funcarglist', ()):
        metafunc.addcall(funcargs=funcargs)

# actual test code, above support code can live elsewhere

class TestClass:
    @params([dict(a=1, b=2), dict(a=3, b=3), dict(a=5, b=4)], )
    def test_equals(self, a, b):
        assert a == b

    @params([dict(a=1, b=0), dict(a=3, b=2)])
    def test_zerodivision(self, a, b):
        py.test.raises(ZeroDivisionError, "a/b")

This variant leaves the "test specification" tightly coupled. Running it with py.test test_parametrize2.py provides the some output as for the first example port.

A quick port of "testscenarios"

Finally, let’s also port Rob Collin’s testscenario example. Here is the implementation of the full mechanism with py.test and the tests in funcarg-style:

#./test_parametrize3.py

def pytest_generate_tests(metafunc):
    for scenario in metafunc.cls.scenarios:
        metafunc.addcall(id=scenario[0], funcargs=scenario[1])

scenario1 = ('basic', {'attribute': 'value'})
scenario2 = ('advanced', {'attribute': 'value2'})

class TestSampleWithScenarios:
    scenarios = [scenario1, scenario2]

    def test_demo(self, attribute):
        assert isinstance(attribute, str)

Let’s run it:


$ py.test -v test_parametrize3.py

================================ test session starts ================================
python: platform linux2 -- Python 2.6.2 -- /usr/bin/python
test object 1: test_parametrize3.py

test_parametrize3.py:14: TestSampleWithScenarios.test_demo[basic] PASS
test_parametrize3.py:14: TestSampleWithScenarios.test_demo[advanced] PASS

============================= 2 passed in 0.06 seconds ==============================

Easy, isn’t it?

Playing yourself

If you want to play with the examples yourself, you can use hg clone https://bitbucket.org/hpk42/py-trunk/ and setup.py install it. In the example/parametrize/ direcrory you can tweak and run the test examples. Let me know of comments or problems you may encounter.

Conclusion: deprecating "yield"

The three ports show that pytest_generate_tests is a hook that allows to implement many custom parametrization schemes. You can implement the hook in a test module or in local or global plugin, sharing it in your project or in the community. The hook also integrates well with other usages of funcargs, see the extensive pytest funcarg documentation.

The new way to parametrize test is meant to substitute yield usage of test-functions aka "generative tests", also used by nosetests. yield-style Generative tests have received criticism and despite being the one who invented them, i mostly agree and recommend not using them anymore.

I’d like to thank Samuele Pedroni and Ronny Pfannschmitt who helped to evolve the new hook and pushed me for implementing it. Oh, and did i emphasize that working feedback-based and documentation driven is so much better than going wild on hypothetical usages?

have fun, holger

Written by holger krekel

May 13, 2009 at 2:55 pm

py.test: shrinks code, grows plugins and integration testing

leave a comment »

Just before Pycon i uploaded the lean and mean py.test 1.0.0b1 beta release. A lot of code got moved out, most notably the greenlets C-extension. This simplifies packaging and increases the py lib’s focus on test facilities. It now has a pluginized architecture and provides funcargs which tremendously help with writing functional and integration tests. One such example are py.test’s own acceptance tests checking behaviour of the command line tool from a user perspective. Other features include a zero-install mechanism for distributing tests which also allow to conveniently drive cross-platform integration tests.

Unittesting, functional and integration testing are now official targets. No doubt, Test category naming is a slippery subject and it’s a good idea to consider test category names as labels rather than "either-or" categories. In the end, tests are about being useful for software development which today means coding for a wide variety of environments and involving integration and deployment issues on every corner. I think that testing tools yet have to develop their full potential. In my opinion, automated testing and deployment techniques are to fully integrate with each other, and i consider coding of distributed integration test scenarios as key to that.

The upcoming final py.test 1.0 release i’d like to make a starting point for facilitating the integration of many more test methods and test mechanisms via plugins. Some people already contributed a pytest_figleaf (for coverage testing) and pytest_twisted (for running twisted style tests) although i am still only finalizing API details and writing up docs. So I am very happy how things are turning out and also motivated by the positive feedback on the two testing tutorials that Brian Dorsey and me gave at Pycon (see his writeup).

Btw, if you use the quickstart and encounter any problems, please use the brand new issue tracker on bitbucket. I started hosting a mercurial py.test trunk repository and so far it’s been a positive experience and I guess I fully switch to mercurial soon. Alas, i probably drop setuptools before i go 1.0 with py.test – it simply causes too many troubles. py.test’s trunk has a straightforward setup.py and i intend to release a second beta with refined docs and removed setuptools. Stay tuned for many more news in May – right now, i am looking forward to some 1-week offline holiday :)

cheers, holger

Written by holger krekel

April 18, 2009 at 9:05 pm

new: simultanously test your code on all platforms

with 5 comments

It is now convenient to do test runs that transparently distribute a normal test run to multiple CPUs or processes. You can do local code changes and immediately distribute tests of your choice simultanously on all available platforms and python versions.

A small example. Suppose you have a Python pkg directory containing the typical __init__.py and a test_something.py test file with this content:

import sys, os

def test_pathsep():
    assert os.sep == "/"

def test_platform():
    assert sys.platform != "darwin"

Without further ado (no special configuration files, no remote installations) i can now run:

py.test pkg/test_something.py --dist=each --rsyncdir=pkg --tx socket=192.168.1.102:8888  --tx ssh=noco --tx popen//python=python2.4

This will rsync the "pkg" directory to the specified test execution places and then run tests on my windows machine (reachable through the specified socket-connection), my mac laptop (reachable through ssh) and in a local python 2.4 process. Here is the full output of the distributed test run which shows 6 tests (4 passed, 2 failed), i.e. our 2 tests multiplied by 3 platforms. It shows one expected failure like so:

[2] ssh=noco -- platform darwin, Python 2.5.1-final-0 cwd: /Users/hpk/pyexecnetcache

    def test_platform():
>       assert sys.platform != "darwin"
E       assert 'darwin' != 'darwin'
E        +  where 'darwin' = sys.platform

/Users/hpk/pyexecnetcache/pkg/test_something.py:8: AssertionError

Hope you find the output obvious enough. I’ve written up some docs in the new distributing tests section.

I also just uploaded a 1.0 alpha py lib release so that you might type "easy_install -U py" to get the alpha release (use at your own risk!). Did i mention that it passes all of its >1000 tests on all platforms simultanously? :)

This is all made possible by py.execnet which is the underlying mechanism for instantiating local and remote processes through SSH- or socket servers and executing code in them, without requiring any prior installation on the remote sides. zero installation really means that you only need a working python interpreter, nothing more. You can make use of this functionality without using py.test. In fact, i plan to soon separate py.test and make the py lib smaller and smaller …

so, enjoy, hope it makes as much sense to you as it makes to me :) And hope to see some of you at Pycon … btw, anybody interested to join me for a Drum and Base party thursday night? cheers, holger

Written by holger krekel

March 23, 2009 at 6:12 pm

Monkeypatching in unit tests, done right

with 12 comments

[updated, thanks to marius].

I am currently preparing my testing tutorials for Pycon and here is an example i’d lke to share already.

The problem: In a test function we want to patch an Environment variable and see if our application handles something related to it correctly. The direct approach for doing this in a test function might look like this:

def test_envreading(self):
    old = os.environ['ENV1']
    os.environ['ENV1'] = 'myval'
    try:
        val = myapp().readenv()
        assert val == "myval"
    finally:
        os.environ['ENV1'] = old

If we needed to do this several times for test functions we’d have a lot of repetetive boilerplatish code. The try-finally and undo-related code does not even take into account that ENV1 might not have been set originally.

Most experienced people would use setup/teardown methods to get less-repetetive testing code. We might end up with something slightly more general like this:

def setup_method(self, method):
    self._oldenv = os.environ.copy()

def teardown_method(self, method):
    os.environ.update(self._oldenv)

def test_envreading(self):
    os.environ['ENV1'] = "myval"
    val = myapp().readenv()
    assert val == "myval"

This avoids repetition of setup code but it scatters what belongs to the test function across three functions. All other functions in the Testcase class will get the service of a preserved environment although they might not need it. If i want to move away this testing function i will need to take care to copy the setup code as well. Or i start subclassing Test cases to share code. If we then start to need modifying other dicts or classes we have to add code in three places.

Monkeypatching the right way

Here is a version of the test function which uses pytest’s monkeypatch` plugin. The plugin does one thing: it provides a monkeypatch object for each test function that needs it. The resulting test function code then looks like this:

def test_envreading(self, monkeypatch):
    monkeypatch.setitem(os.environ, 'ENV1', 'myval')
    val = myapp().readenv()
    assert val == "myval"

Here monkeypatch.setitem() will memorize old settings and modify the environment. When the test function finishes the monkeypatch object restores the original setting. This test function is free to get moved across files. No other test function or code place is affected or required to change when it moves.

Let’s take a quick look at the "providing" side, i.e. the pytest_monkeypatch.py plugin which provides "Monkeypatch" instances to test functions. It makes use of pytest’s new pyfuncarg protocol.

The plugin itself is free to get refined and changed as well, without affecting the existing test code. The following 71 lines of code make up the plugin, including tests:

class MonkeypatchPlugin:
    """ setattr-monkeypatching with automatical reversal after test. """
    def pytest_pyfuncarg_monkeypatch(self, pyfuncitem):
        monkeypatch = MonkeyPatch()
        pyfuncitem.addfinalizer(monkeypatch.finalize)
        return monkeypatch

notset = object()

class MonkeyPatch:
    def __init__(self):
        self._setattr = []
        self._setitem = []

    def setattr(self, obj, name, value):
        self._setattr.insert(0, (obj, name, getattr(obj, name, notset)))
        setattr(obj, name, value)

    def setitem(self, dictionary, name, value):
        self._setitem.insert(0, (dictionary, name, dictionary.get(name, notset)))
        dictionary[name] = value

    def finalize(self):
        for obj, name, value in self._setattr:
            if value is not notset:
                setattr(obj, name, value)
            else:
                delattr(obj, name)
        for dictionary, name, value in self._setitem:
            if value is notset:
                del dictionary[name]
            else:
                dictionary[name] = value


def test_setattr():
    class A:
        x = 1
    monkeypatch = MonkeyPatch()
    monkeypatch.setattr(A, 'x', 2)
    assert A.x == 2
    monkeypatch.setattr(A, 'x', 3)
    assert A.x == 3
    monkeypatch.finalize()
    assert A.x == 1

    monkeypatch.setattr(A, 'y', 3)
    assert A.y == 3
    monkeypatch.finalize()
    assert not hasattr(A, 'y')


def test_setitem():
    d = {'x': 1}
    monkeypatch = MonkeyPatch()
    monkeypatch.setitem(d, 'x', 2)
    monkeypatch.setitem(d, 'y', 1700)
    assert d['x'] == 2
    assert d['y'] == 1700
    monkeypatch.setitem(d, 'x', 3)
    assert d['x'] == 3
    monkeypatch.finalize()
    assert d['x'] == 1
    assert 'y' not in d

def test_monkeypatch_plugin(testdir):
    sorter = testdir.inline_runsource("""
        pytest_plugins = 'pytest_monkeypatch',
        def test_method(monkeypatch):
            assert monkeypatch.__class__.__name__ == "MonkeyPatch"
    """)
    res = sorter.countoutcomes()
    assert tuple(res) == (1, 0, 0), res

I can also imagine some nice plugin which supports mock objects – patching methods with some preset behaviour or tracing calls between components.

have fun, holger

Written by holger krekel

March 3, 2009 at 1:48 pm

New Plugin architecture and plugins for py.test

with one comment

I just merged the plugin branch and am very happy about it. Part of the effort was driven by moving core functionality to become a plugin: Terminal reporting is now fully a plugin, contained in a single file including tests. It does it work solely by looking at testing events. Plugins can also add new aspects to tests files – for example the pytest_restdoc.py plugin adds ReST syntax, referential integrity and URL checking for Text files. (I used it for checking my blog post and its links, btw).

Pytest’s good old conftest.py files are still useful: you can define project or directory specific settings, including which plugins to use. For now, many old extensions should work unmodified, as exemplified by PyPy‘s extensive conftest.py files. It’s easy to port a conftest file to a plugin. In fact, you can first define a local "ConftestPlugin" and later move it to become a cross-project one – a matter of renaming the file and the class, done!

To serve as guiding examples, I drafted some initial plugins and implemented neccessary hooks within py.test core.

If you wan’t to get a feel on how plugins are implemented, here is the pytest_eventlog.py plugin which adds a command line option to allow logging of all testing events. It’s instructive to look at how it’s done as well as the output because it shows which testing events are generated.

class EventlogPlugin:
    """ log pytest events to a file. """

    def pytest_addoption(self, parser):
        parser.addoption("--eventlog", dest="eventlog",
            help="write all pytest events to the given file.")

    def pytest_configure(self, config):
        eventlog = config.getvalue('eventlog')
        if eventlog:
            self.eventlogfile = open(eventlog).open('w')

    def pytest_unconfigure(self, config):
        if hasattr(self, 'eventlogfile'):
            self.eventlogfile.close()
            del self.eventlogfile

    def pyevent(self, eventname, *args, **kwargs):
        if hasattr(self, 'eventlogfile'):
            print >>self.eventlogfile, eventname, args, kwargs
            self.eventlogfile.flush()

This plugin code is complete, except that the original pytest_eventlog.py file contains tests. The eventlog plugin methods above are called in the following way:

  • def pytest_addoption(self, parser) is called before
    commandline arguments are parsed.
  • def pytest_configure(self, config) is called after parsing
    arguments and before any reporting, collection or running
    of tests takes place.
  • def pytest_event(self, eventname, *args, **kwargs) is called
    for each testing event. Events have names and come with
    arguments which are supplied by the event producing site.
  • def pytest_unconfigure(self, config) is called after
    all test items have been processed.

If you want to start writing your own plugin, please use an svn checkout of:

http://codespeak.net/svn/py/trunk/

and activate it by e.g. python setup.py develop.

If you want to write a plugin named pytest_XYZ, you can tell pytest to use it by setting the environment variable PYTEST_PLUGINS=XYZ or by putting pytest_plugins = 'xyz' into a test module or conftest.py file.

A good way to contribute is to copy an existing plugin file to your home dir and put it somewhere into your PYTHONPATH. py.test will use your version instead of the default one and you can play with it untill you are happy (and see to also add some tests showing the new behaviour).

If you have questions or problems, you are invited to post here or to the py-dev mailing list. I’d definitely like to pluginize more of pytest and add hooks as needed and am happy for feedback and suggestions before i freeze the API for 1.0.

holger

Written by holger krekel

February 27, 2009 at 11:22 am

New way to organize Python test code

with 2 comments

py.test just grew a new way to provide test state for a test function. First the problem: those of us dealing with writing tests in the hundreds or thousands usually setup test state at class, method or module level. Then we access it indirectly, through self, local or global helper functions. For larger applications, this usually leads to scattered, complex and boilerplatisch test code. This then stands in the way of refactoring the original code base … but wait, weren’t tests meant to ease refactoring, not hinder it?

Here is the idea: Python Test functions use their function definition to state their needs and the test tools calls a function that provides the value. For example, consider this test function:


   def test_ospath(self, tempdir):
      # needs tempdir to create files etc.

.

py.test provides the value for tempdir by calling a matching method that looks like this:


  def pytest_pyfuncarg_tempdir(pyfuncitem):
      # use pyfuncitem to access test context, cmdline opts etc.

.

This matching provider function returns a value for tempdir that is then supplied to the test function. For more complex purposes, the pyfuncitem argument provides full access to the test collection process including cmdline options, test options, project specific configuration. You can write down this provider method in the test
module, in configuration files or in a plugin.

Once i started using this new paradigm, i couldn’t resist and refactored pytest’s own tests to use the new method everywhere. Here are my findings so far:

  • self contained test functions: i don’t need to wade through unneccessary layers and indirection of test setup.
  • fewer imports: my test modules don’t need to import modules that are only needed for setting up application state.
  • easy test state setup: I can place test support code in one place and i can grep for pytest_pyfuncarg_NAME. I can reuse this setup code easily across modules, directories or even projects. Think about providing test database object or mocking objects.
  • more flexible test grouping: I can logically group tests however i like, independently from test setup requirements. I found it very easy to shuffle test functions between classes or modules because they are rather self-contained.

Written by holger krekel

February 22, 2009 at 5:23 pm

Follow

Get every new post delivered to your Inbox.