metaprogramming and politics

Time for a bit of fiction on distributed Python deployment. As some of you know, py.execnet imperatively and elastically executes code in local or remote python processes, maintaining channels for exchanging data. Execnet has the wonderful zero-install feature which means no software except a Python interpreter is required remotely. The connection between Python interpreters is direct, i.e. the connecting side needs to know how to start the remote side. And it’s non-transitive meaning: given a A->B and a B->C connection there is no support for getting a A->C connection mediated by B.

I’d like to lift these restriction and introduce the concept of a deployment network through which execnet connections can be mediated. I am pondering a command line tool that creates a network of Python intepreters on multiple hosts like this:

execnet start mynet ssh=linuxbox.org socket=windowsbox.com ssh=osx.com

This would create a "mynet" deployment network of four Python interpreters running on different hosts and platforms: one local process and three remote processes connected to it. Let’s add a remote Jython process to the "mynet" deployment network:

execnet addhost mynet ssh=remote//python=jython

We can generally use the ‘mynet’ handle to work with this newly instantiated deployment network. For example, to get a fresh process on a certain platform from a Python program:

mynet = execnet.connect('mynet')
gateway = mynet.makegateway(platform="java")

The first line connects to our local ‘mynet’ process. The second line creates a gateway to a fresh Python interpreter, in this case a Jython process. The bootstrapping of the Jython-side gateway object is determined by the initiating client side. The two subprocesses communicate through an IO-connection that is mediated by the ‘mynet’ deployment network.

This is very exciting because the zero-installation feature is preserved on two levels: the deployment processes work on software coming from a single point, the command line above. And our "on-top" gateways operate with software determined from the initiating side, from the python code above. Interaction between the two worlds is limited to a connect operation and providing IO mediatiation. This means the deployment network facilities can evolve independently from the "on-top" execnet-elastic programs.

Conceptually it’s a very reliable and robust setting. The mynet processes should be able to run as robustly as unix shells. They are to provide a solid base for writing and deploying Python applications that span multiple interpreters. They don’t run applications in-process.

This is not all fiction. The current development version of py.execnet already works across and between CPython2.4 through to CPython 3.1, Jython and PyPy. And i intend to release execnet as a separate package soon, providing the basis for implementing the above fiction and lots of other on-top fun 🙂