New way to organize Python test code
py.test just grew a new way to provide test state for a test function. First the problem: those of us dealing with writing tests in the hundreds or thousands usually setup test state at class, method or module level. Then we access it indirectly, through self, local or global helper functions. For larger applications, this usually leads to scattered, complex and boilerplatisch test code. This then stands in the way of refactoring the original code base … but wait, weren’t tests meant to ease refactoring, not hinder it?
Here is the idea: Python Test functions use their function definition to state their needs and the test tools calls a function that provides the value. For example, consider this test function:
def test_ospath(self, tempdir):
# needs tempdir to create files etc.
.
py.test provides the value for tempdir by calling a matching method that looks like this:
def pytest_pyfuncarg_tempdir(pyfuncitem):
# use pyfuncitem to access test context, cmdline opts etc.
.
This matching provider function returns a value for tempdir that is then supplied to the test function. For more complex purposes, the pyfuncitem argument provides full access to the test collection process including cmdline options, test options, project specific configuration. You can write down this provider method in the test
module, in configuration files or in a plugin.
Once i started using this new paradigm, i couldn’t resist and refactored pytest’s own tests to use the new method everywhere. Here are my findings so far:
- self contained test functions: i don’t need to wade through unneccessary layers and indirection of test setup.
- fewer imports: my test modules don’t need to import modules that are only needed for setting up application state.
- easy test state setup: I can place test support code in one place and i can grep for pytest_pyfuncarg_NAME. I can reuse this setup code easily across modules, directories or even projects. Think about providing test database object or mocking objects.
- more flexible test grouping: I can logically group tests however i like, independently from test setup requirements. I found it very easy to shuffle test functions between classes or modules because they are rather self-contained.
Written by holger krekel
February 22, 2009 at 5:23 pm
Posted in metaprogramming, testing
5 Responses
Subscribe to comments with RSS.
metaprogramming and politics 
Great post. I would suggest you post about the new plugin architecture and comparisons with Trail and Nose.
Leonardo Santagada
February 22, 2009 at 9:21 pm
[…] Let’s take a quick look at the "providing" side, i.e. the pytest_monkeypatch.py plugin which provides "Monkeypatch" instances to test functions. It makes use of pytest’s new pyfuncarg protocol. […]
Monkeypatching in unit tests, done right « Tetamap: on metaprogramming, Python and testing
March 3, 2009 at 1:48 pm
Hmmm, this is interesting. I’m just getting to grips with a codebase that uses pytest for the first time. I can see this is a handy technique, I do like tests that are self-contained. My previous method of achieving this is to have the test call setup functions during test setup. I’m currently torn between the brevity of the pyfuncarg method, versus the explicit lack of magic in simply calling a function. It’s true the functions need more imports and to verbosely call test setup code. But on the other hand, coming to it fresh, I had to come read documentation and blog posts like this to find out where the heck ‘monkeypatch’ was coming from, and I still don’t know how to use it in my existing unittest style test classes – which would have been self evident if the default way of using it was just “from pytest import monkeypatch” No doubt more will become clear as I read more.
I trust your judgement, Holger: If you have a moment, do you still like this style after some years of experience with it?
Jonathan Hartley
July 15, 2015 at 7:46 am
It’s true that the fact that fixtures come from conftest files and 3rd party plugins instead of imports makes them harder to discover. If you run “py.test –fixtures“ you should the information however.
I still like the fixture system and use it everywhere. I think that if you look around on twitter or blog posts you’ll find that many people have adapted them, especially the “yield_fixture“ style.
Then again, i am biased as i was heavily involved in coming up with the idea đŸ™‚
holger krekel
July 19, 2015 at 2:44 pm
Fair enough. In that case I’ll continue diving in without reservation. Thanks!
tartley
July 23, 2015 at 7:57 pm