new: simultanously test your code on all platforms
It is now convenient to do test runs that transparently distribute a normal test run to multiple CPUs or processes. You can do local code changes and immediately distribute tests of your choice simultanously on all available platforms and python versions.
A small example. Suppose you have a Python pkg directory containing the typical __init__.py and a test_something.py test file with this content:
import sys, os
def test_pathsep():
assert os.sep == "/"
def test_platform():
assert sys.platform != "darwin"
Without further ado (no special configuration files, no remote installations) i can now run:
py.test pkg/test_something.py --dist=each --rsyncdir=pkg --tx socket=192.168.1.102:8888 --tx ssh=noco --tx popen//python=python2.4
This will rsync the "pkg" directory to the specified test execution places and then run tests on my windows machine (reachable through the specified socket-connection), my mac laptop (reachable through ssh) and in a local python 2.4 process. Here is the full output of the distributed test run which shows 6 tests (4 passed, 2 failed), i.e. our 2 tests multiplied by 3 platforms. It shows one expected failure like so:
[2] ssh=noco -- platform darwin, Python 2.5.1-final-0 cwd: /Users/hpk/pyexecnetcache
def test_platform():
> assert sys.platform != "darwin"
E assert 'darwin' != 'darwin'
E + where 'darwin' = sys.platform
/Users/hpk/pyexecnetcache/pkg/test_something.py:8: AssertionError
Hope you find the output obvious enough. I’ve written up some docs in the new distributing tests section.
I also just uploaded a 1.0 alpha py lib release so that you might type "easy_install -U py" to get the alpha release (use at your own risk!). Did i mention that it passes all of its >1000 tests on all platforms simultanously? 🙂
This is all made possible by py.execnet which is the underlying mechanism for instantiating local and remote processes through SSH- or socket servers and executing code in them, without requiring any prior installation on the remote sides. zero installation really means that you only need a working python interpreter, nothing more. You can make use of this functionality without using py.test. In fact, i plan to soon separate py.test and make the py lib smaller and smaller …
so, enjoy, hope it makes as much sense to you as it makes to me 🙂 And hope to see some of you at Pycon … btw, anybody interested to join me for a Drum and Base party thursday night? cheers, holger
Written by holger krekel
March 23, 2009 at 6:12 pm
Posted in metaprogramming, testing
7 Responses
Subscribe to comments with RSS.
metaprogramming and politics 
py.execnet looks very interesting, but also quite frightening. “Execution of any python code on a remote platform”, that opens lots of doors for root exploits.
Bluebird
March 24, 2009 at 6:48 am
Hello, sorry for taking so long, am enjoying a quite busy Pycon US conference.
I’d like to offer a couple of considerations regarding the security concern:
* for distributed testing purposes executing code on the system-under-test is probably not a concern, because it’s the developer sending his code to the test environments and it is obviously his own code and he usually has control over the test environment.
* with py.execnet your code can distribute parts of itself to other CPUs or hosts. Doing an install of your code manually does not change the fact it is your code that will be running there. so i’d argue that for distributing your code, be it statically/manually or dynamically/zero-install like, the basic same security considerations apply.
* if the “remote side” does not trust your python code execution, it should run the code in some sandboxing or virtualized environment, either on a whole system level or by using a virtualized Python interpreter like PyPy’s sandboxing.
hope that makes sense to you,
holger
holger krekel
March 29, 2009 at 2:33 pm
[…] py.test beta1: shrinks code, grows plugins, integration testing! Posted in python by holger krekel on April 18, 2009 Just before Pycon i uploaded the lean and mean py.test 1.0.0b1 beta release. A lot of code got moved out, most notably the greenlets C-extension. This simplifies packaging and increases the py lib’s focus on test facilities. It now has a pluginized architecture and provides funcargs which tremendously help with writing functional and integration tests. One such example are py.test’s own acceptance tests checking behaviour of the command line tool from a user perspective. Other features include a zero-install mechanism for distributing tests which also allow to conveniently drive cross-platform integration tests. […]
py.test beta1: shrinks code, grows plugins, integration testing! « Tetamap: on metaprogramming, Python and testing
April 18, 2009 at 9:05 pm
I am having a bear or a time getting distributed testing to work. I hope you can offer some advice. First, a question and then onto my biggest issue.
In my case, I have a second set of tests I want to run on another machine. I figure I can use –dist=each and do a hostname check in the scripts to make sure the bulk of the script runs only on the right machine. Is there a better way?
My big problem is that I can’t get the socketserver.py to work right on the remote machine. Here is my command line on the main server:
c:\Python26\Scripts\py.test –dist=each –tx socket=192.168.1.11:8888 tests\test_guest_install.py
If I read the doc right, this should push the script to the remote machine and run it. Here’s the output on the other side:
C:\Users\Alan\Desktop>c:\Python26\python socketserver.py
socket_readline_exec_server-1.2 Entering Accept loop (‘0.0.0.0’, 8888)
socket_readline_exec_server-1.2 got new connection from 192.168.1.5 30856
reading line
socket_readline_exec_server-1.2 compiled source, executing
============================= test session starts =============================
python: platform win32 — Python 2.6.4 — pytest-1.2.0
test object 1: C:\Users\Alan\Desktop\pyexecnetcache
============================== in 0.14 seconds ===============================
socket_readline_exec_server-1.2 finished executing code
leaving socketserver execloop
Traceback (most recent call last):
File “socketserver.py”, line 90, in
startserver(serversock, loop=False)
File “socketserver.py”, line 81, in startserver
serversock.shutdown(2)
File “”, line 1, in shutdown
socket.error: [Errno 10057] A request to send or receive data was disallowed be
ause the socket is not connected and (when sending on a datagram socket using a
sendto call) no address was supplied
It doesn’t look like I get the result back on the main machine. If you could shed some light on what I am doing wrong, I would be thankful.
Alan Keister
February 2, 2010 at 2:34 am
ufa
mAmuli polo a tafu
July 28, 2020 at 10:48 am
Hi Alan,
regarding sending tests to a particular machine:
currently tests cannot specify a particular target machine to run on. I think this would be useful and straight forward to implement – e.g. we could use a decorator like @py.test.mark.tx(id=”xyz”)
and have the xdist plugin care for only sending the test to the particular test node … do you mind bringing up this particular questions on the py-dev mailing list? Is a bit more comfortable than working in the wordpress text box 🙂
as to the other issue with the socketserver, i am not sure what the root problem is. Do you use execnet-1.0.4 and the socketserver.py from it?
Can you try to run things completely on localhost?
Feel free to mail me at holger at merlinux eu about this as well – i’d reply then to the mailing list as well if you don’t mind.
cheers & thanks,
holger
holger krekel
February 2, 2010 at 9:18 am
Hello.. I am using Spyder (Python 3.5), Ipython console.. How do I test a function using pytest ?
Following is function I need to test
def test_three_steps():
from math import pi
I = 1; w = 2*pi; num_periods = 5
P = 2*np.pi/w # one period
T = P*num_periods
dt = 0.05
u_by_hand = np.array([1.000000000000000,0.802607911978213, 0.288358920740053])
u, t = solver(I, w, dt, T)
diff = np.abs(u_by_hand – u[:3]).max()
tol = 1E-14
assert diff < tol
Nikhil Yewale
March 14, 2017 at 8:06 am