metaprogramming and politics

Decentralize. Take the red pill.

new: simultanously test your code on all platforms

with 6 comments

It is now convenient to do test runs that transparently distribute a normal test run to multiple CPUs or processes. You can do local code changes and immediately distribute tests of your choice simultanously on all available platforms and python versions.

A small example. Suppose you have a Python pkg directory containing the typical and a test file with this content:

import sys, os

def test_pathsep():
    assert os.sep == "/"

def test_platform():
    assert sys.platform != "darwin"

Without further ado (no special configuration files, no remote installations) i can now run:

py.test pkg/ --dist=each --rsyncdir=pkg --tx socket=  --tx ssh=noco --tx popen//python=python2.4

This will rsync the "pkg" directory to the specified test execution places and then run tests on my windows machine (reachable through the specified socket-connection), my mac laptop (reachable through ssh) and in a local python 2.4 process. Here is the full output of the distributed test run which shows 6 tests (4 passed, 2 failed), i.e. our 2 tests multiplied by 3 platforms. It shows one expected failure like so:

[2] ssh=noco -- platform darwin, Python 2.5.1-final-0 cwd: /Users/hpk/pyexecnetcache

    def test_platform():
>       assert sys.platform != "darwin"
E       assert 'darwin' != 'darwin'
E        +  where 'darwin' = sys.platform

/Users/hpk/pyexecnetcache/pkg/ AssertionError

Hope you find the output obvious enough. I’ve written up some docs in the new distributing tests section.

I also just uploaded a 1.0 alpha py lib release so that you might type "easy_install -U py" to get the alpha release (use at your own risk!). Did i mention that it passes all of its >1000 tests on all platforms simultanously? 🙂

This is all made possible by py.execnet which is the underlying mechanism for instantiating local and remote processes through SSH- or socket servers and executing code in them, without requiring any prior installation on the remote sides. zero installation really means that you only need a working python interpreter, nothing more. You can make use of this functionality without using py.test. In fact, i plan to soon separate py.test and make the py lib smaller and smaller …

so, enjoy, hope it makes as much sense to you as it makes to me 🙂 And hope to see some of you at Pycon … btw, anybody interested to join me for a Drum and Base party thursday night? cheers, holger

Written by holger krekel

March 23, 2009 at 6:12 pm

6 Responses

Subscribe to comments with RSS.

  1. py.execnet looks very interesting, but also quite frightening. “Execution of any python code on a remote platform”, that opens lots of doors for root exploits.


    March 24, 2009 at 6:48 am

  2. Hello, sorry for taking so long, am enjoying a quite busy Pycon US conference.

    I’d like to offer a couple of considerations regarding the security concern:

    * for distributed testing purposes executing code on the system-under-test is probably not a concern, because it’s the developer sending his code to the test environments and it is obviously his own code and he usually has control over the test environment.

    * with py.execnet your code can distribute parts of itself to other CPUs or hosts. Doing an install of your code manually does not change the fact it is your code that will be running there. so i’d argue that for distributing your code, be it statically/manually or dynamically/zero-install like, the basic same security considerations apply.

    * if the “remote side” does not trust your python code execution, it should run the code in some sandboxing or virtualized environment, either on a whole system level or by using a virtualized Python interpreter like PyPy’s sandboxing.

    hope that makes sense to you,

    holger krekel

    March 29, 2009 at 2:33 pm

  3. […] py.test beta1: shrinks code, grows plugins, integration testing! Posted in python by holger krekel on April 18, 2009 Just before Pycon i uploaded the lean and mean py.test 1.0.0b1 beta release. A lot of code got moved out, most notably the greenlets C-extension. This simplifies packaging and increases the py lib’s focus on test facilities. It now has a pluginized architecture and provides funcargs which tremendously help with writing functional and integration tests. One such example are py.test’s own acceptance tests checking behaviour of the command line tool from a user perspective. Other features include a zero-install mechanism for distributing tests which also allow to conveniently drive cross-platform integration tests. […]

    • I am having a bear or a time getting distributed testing to work. I hope you can offer some advice. First, a question and then onto my biggest issue.

      In my case, I have a second set of tests I want to run on another machine. I figure I can use –dist=each and do a hostname check in the scripts to make sure the bulk of the script runs only on the right machine. Is there a better way?

      My big problem is that I can’t get the to work right on the remote machine. Here is my command line on the main server:

      c:\Python26\Scripts\py.test –dist=each –tx socket= tests\

      If I read the doc right, this should push the script to the remote machine and run it. Here’s the output on the other side:

      socket_readline_exec_server-1.2 Entering Accept loop (‘’, 8888)
      socket_readline_exec_server-1.2 got new connection from 30856
      reading line
      socket_readline_exec_server-1.2 compiled source, executing
      ============================= test session starts =============================
      python: platform win32 — Python 2.6.4 — pytest-1.2.0
      test object 1: C:\Users\Alan\Desktop\pyexecnetcache

      ============================== in 0.14 seconds ===============================
      socket_readline_exec_server-1.2 finished executing code
      leaving socketserver execloop
      Traceback (most recent call last):
      File “”, line 90, in
      startserver(serversock, loop=False)
      File “”, line 81, in startserver
      File “”, line 1, in shutdown
      socket.error: [Errno 10057] A request to send or receive data was disallowed be
      ause the socket is not connected and (when sending on a datagram socket using a
      sendto call) no address was supplied

      It doesn’t look like I get the result back on the main machine. If you could shed some light on what I am doing wrong, I would be thankful.

      Alan Keister

      February 2, 2010 at 2:34 am

  4. Hi Alan,

    regarding sending tests to a particular machine:
    currently tests cannot specify a particular target machine to run on. I think this would be useful and straight forward to implement – e.g. we could use a decorator like @py.test.mark.tx(id=”xyz”)
    and have the xdist plugin care for only sending the test to the particular test node … do you mind bringing up this particular questions on the py-dev mailing list? Is a bit more comfortable than working in the wordpress text box 🙂

    as to the other issue with the socketserver, i am not sure what the root problem is. Do you use execnet-1.0.4 and the from it?
    Can you try to run things completely on localhost?

    Feel free to mail me at holger at merlinux eu about this as well – i’d reply then to the mailing list as well if you don’t mind.

    cheers & thanks,

    holger krekel

    February 2, 2010 at 9:18 am

  5. Hello.. I am using Spyder (Python 3.5), Ipython console.. How do I test a function using pytest ?
    Following is function I need to test

    def test_three_steps():
    from math import pi
    I = 1; w = 2*pi; num_periods = 5
    P = 2*np.pi/w # one period
    T = P*num_periods
    dt = 0.05
    u_by_hand = np.array([1.000000000000000,0.802607911978213, 0.288358920740053])
    u, t = solver(I, w, dt, T)
    diff = np.abs(u_by_hand – u[:3]).max()
    tol = 1E-14
    assert diff < tol

    Nikhil Yewale

    March 14, 2017 at 8:06 am

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: