metaprogramming and politics

Decentralize. Take the red pill.

Posts Tagged ‘metaprogramming

Packaging, testing, pypi and my Pycon Russia adventures

leave a comment »

A few days ago I talked at Pycon Russia on packaging and testing and a new PyPI Server implementation and workflow tool i am working on, codenamed devpi. See the slides and the video. The slides are converted from my hovercraft based presentation which you can find here (needs javascript).  devpi tries to solve the “standardization” problem around Python packaging by offering a good index server and a “meta approach” on configuring and invoking setup.py/easy_install/pip, incorporating existing practises and facilitating new ones.  The slides and the talk hopefully clarify a bit of the reasoning behind it.
Besides the good feedback and discussions around my talk, i just had a great few days. It was my first time to Russia and i saw and learned a lot.  One unexpected event was going to a russian Sauna with Amir Salihefendic, Russel Keith-Magee and Anton, a main conference organizer. Between going into the Sauna we had glasses of nice irish Whiskey or walked outside to the snowy freezing cold.  Afterwards some of us went to the conference party and had good (despite being somewhat drunken) discussions with people from Yandex, the biggest russian search engine and several russian devs. All very friendly, competent and funny. The party lasted until 5:30am – with my fellow english-speaking talkers Armin Ronacher, David Cramer (a weekend in Russia) and me being among the very last.

Image

David, Amir, Russel, and our russian hosts

The next days evening saw Amir, David, Armin and two russian guys visiting an Irish pub past midnight. It turned out there is no such thing as a “russian pub”, the concept of “pub” was imported in the last decade mostly in the form of english or irish ones. And it seems IT/Python guys can meet everywhere on the planet and have a good time :)

Image

Ice, Ekaterinburg at night, and an anonymous shop

Going back to content, i felt particularly inspired by Jeff Lindsay’s talk on Autosustainable services. He described how he tries to provide several small web services, and how to organize cost sharing by its users. As services need resources, it’s a different issue than Open Cource collaboration which does not require such to exist.

I heart several good sentences from my fellow talkers, for example one from Russel Keith-Magee describing a dillema of open source communities: “There are many people who can say ‘No’ but few who can say ‘Yes’ to something”. Amir Salihefendic desribed how the “Redis” database solved many problems for him, and some interesting concrete usages of “bitmaps” in his current endeavours like bitmapist.cohort. And of course Armin Ronacher and David Cramer also gave good talks related to their experience, Advanced Flask patterns and scalable web services respectively.  With Armin i also had a good private discussion about the issue of code-signing and verification.  We drafted what we think could work for Python packaging (more separately).  With David, i discussed workflow commands for python packaging as he offered some good thoughts on the matter.

Around the whole conference we were warmly cared for by Yulia’s company it-people.ru who overtook the physical organisation, and by Anton and his friends who organized the program.  Maria Kalinina in particular had cared for the keynote speakers and many other aspects of the conference, and without her, i wouldn’t have made it.  Anton drove us to the Asian European geographic border, and Yulia to the skyscraper of Ekaterinburg, overlooking the third largest city in Russia. Russel and me also took the opportunity to walk around Ekaterinburg, looking at Lenin sculptures, buildings made of ice, frozen lakes, and the many shops and noises in the city.

Image

Iced lake, Lenin forver, The Asia/Europe border

Lastly i went to the university with Russel to talk for two hours to students about “How Open Source can help your career” and we had a lively discussion with them and the lecturer who invited us.  I offered my own background and stated that the very best people in the IT world are today collaborating through open-source.  It’s a totally dominant model for excellence.  (Which doesn’t mean there are not some good proprietary projects, they are just fewer i’d say).

So i can join the many russian participants who thought Pycon Russia was a very good conference. It’s of course mostly interesting for people speaking russian, as only seven talks were in english.   For my part, the intense time i had with both the russian hosts and developers and the english talkers was verymuch worth it – i think there might be a few new collaborations coming from it.  More on that in later blog posts hopefully  :)

Two days ago i left Ekaterinburg and felt a bit sad because of the many contacts i made, which almost felt like the beginning of friendships.

Written by holger krekel

March 1, 2013 at 12:38 pm

metaprogramming in Python: What CPython, PyPy, Pyramid, pytest and politics have in common …

leave a comment »

Metaprogramming in Python too often revolves around metaclasses, which are just a narrow application of the “meta” idea and not a great one at that. Metaprogramming more generally deals with reasoning about program code, about taking a “meta” stance on it.  A metaprogram takes a program as input, often just partial programs like functions or classes. Here are a few applications of metaprogramming:

  • CPython is a metaprogram written in C. It takes Python program code as input and interprets it, so that it runs at a higher level than C.
  • PyPy is a metaprogramm written in Python. It takes RPython program code as input and generates a C-level metaprogram (the PyPy interpreter) which itself interprets Python programs and takes another meta stance by generating Assembler pieces for parts of the interpreation execution. If you like, PyPy is a metaprogram generating metaprograms whereas CPython and typical compilers like GCC are “just” a metaprogram.
  • Pyramid is a metaprogram that takes view, model definitions and http-handling code as input and executes them, thereby raising code on a higher level to implement the “Pyramid application” language.
  • pytest is a metaprogram written in Python, taking test, fixture and plugin functions as input and executing them in a certain manner, thereby implementing a testing language.
  • metaclasses: in Python they allow to intercept class creation and introspect methods and attributes, amending their behaviour. Because metaclass-code usually executes at import time, it often uses global state for implementing non-trivial meta aspects.

Apart from these concrete examples, language compilers, testing tools and web frameworks all have metaprogramming aspects. Creating big or small “higher” level or domain-specific languages within Python is as a typical example of metaprogramming. Python is actually a great language for metaprogramming although it could be better.

In future blog posts i plan to talk about some good metaprogramming practise, particularly:

  • keep the layers/levels separate by good naming and API design
  • define a concise “language” for the programs you take as input
  • avoid creating global state in your metaprograms (and elsewhere)
    which can easily happen with meta-classes executing at import time

Lastly, i see metaprogramming at work not only when coding in a computer language. Discussing the legal framing for executing programs on the internet is some kind of metaprogramming, especially if you consider licensing and laws as human-interpreted code which affects how programs can be written, constructed and executed. In reverse, web applications increasingly affect how we interact with each other other, thereby implementing rules formerly dealt with in the arena of politics. Therefore, metaprogramming and politics are fundamentally connected topics.

have metafun, i. e. take fun stuff as input to generate more of it :) holger

Written by holger krekel

November 22, 2012 at 3:04 pm

execution locals: better than thread locals/globals

with 6 comments

While many agree that global state is evil, the so called “thread locals” are not much better. Even though they help to separate state on a per-thread or per-greenlet basis, they still are global within that context. In particular (thread) global state means that:

  • Invoked functions can change bindings of an invoking function as a side effect
  • thread locals may linger around even if their state is not used or became invalid

Meet “execution locals” which avoid these problems. Find the code released on PyPI:

http://pypi.python.org/pypi/xlocal

It’s some 60 lines of code and tested on python2.5 up to python3.3 and pypy and ready to be played with. I inline its README.txt below in case you can’t or don’t want to switch reading context. One more note: If I were to design a new language i’d probably remove “globals” all together and only offer something like the “xlocal” type with a more straight forward syntax.

execution locals: killing global state (including thread locals)

The xlocal module provides execution locals aka “xlocal” objects which implement a more restricted variant of “thread locals”. An “xlocal” instance allows to manage its attributes on a per-execution basis in a manner similar to how real locals work:

  • Invoked functions cannot change the binding for the invoking function
  • existence of a binding is local to a code block (and everything it calls)

Attribute bindings for an xlocal object will not leak outside a context-managed code block and they will not leak to other threads or greenlets. By contrast, both process-globals and “thread locals” do not implement these properties.

Let’s look at a basic example:

# content of example.py

from xlocal import xlocal

xcurrent = xlocal()

def output():
    print "hello world", xcurrent.x

if __name__ == "__main__":
    with xcurrent(x=1):
        output()

If we execute this module, the output() function will see a xcurrent.x==1 binding:

$ python example.py
hello world 1

Here is what happens in detail: xcurrent(x=1) returns a context manager which sets/resets the x attribute on the xcurrent object. While remaining in the same thread/greenlet, all code triggered by the with-body (in this case just the output() function) can access xcurrent.x. Outside the with- body xcurrent.x would raise an AttributeError. It is also not allowed to directly set xcurrent attributes; you always have to explicitely mark their life-cycle with a with-statement. This means that invoked code:

  • cannot rebind xlocal state of its invoking functions (no side effects, yay!)
  • xlocal state does not leak outside the with-context (lifecylcle control)

Another module may now reuse the example code:

# content of example_call.py
import example

with example.xcurrent(x=3):
    example.output()

which when running …:

$ python example_call.py
hello world 3

will cause the example.output() function to print the xcurrent.x binding as defined at the invoking with xcurrent(x=3) statement.

Other threads or greenlets will never see this xcurrent.x binding; they may even set and read their own distincit xcurrent.x object. This means that all threads/greenlets can concurrently call into a function which will always see the execution specific x attribute.

Usage in frameworks and libraries invoking “handlers”

When invoking plugin code or handler code to perform work, you may not want to pass around all state that might ever be needed. Instead of using a global or thread local you can safely pass around such state in execution locals. Here is a pseudo example:

xcurrent = xlocal()

def with_xlocal(func, **kwargs):
    with xcurrent(**kwargs):
        func()

def handle_request(request):
    func = gethandler(request)  # some user code
    spawn(with_xlocal(func, request=request))

handle_request will run a user-provided handler function in a newly spawned execution unit (for example spawn might map to threading.Thread(…).start() or to gevent.spawn(…)). The generic with_xlocal helper wraps the execution of the handler function so that it will see a xcurrent.request binding. Multiple spawns may execute concurrently and xcurrent.request will carry the execution-specific request object in each of them.

Issues worth noting

If a method decides to memorize an attribute of an execution local, for example the above xcurrent.request, then it will keep a reference to the exact request object, not the per-execution one. If you want to keep a per-execution local, you can do it this way for example:

Class Renderer:
    @property
    def request(self):
        return xcurrent.request

this means that Renderer instances will have an execution-local self.request object even if the life-cycle of the instance crosses execution units.

Another issue is that if you spawn new execution units, they will not implicitely inherit execution locals. Instead you have to wrap your spawning function to explicitely set execution locals, similar to what we did in the above “invoking handlers” section.

Written by holger krekel

November 16, 2012 at 2:22 pm

pylib 1.0.0 released: the testing-with-python innovations continue

with 5 comments

Took a few betas but finally i uploaded a 1.0.0 py lib release, featuring the mature and powerful py.test tool and "execnet-style" elastic distributed programming. With the new release, there are many new advanced automated testing features – here is a quick summary:

  • funcargs – pythonic zero-boilerplate fixtures for Python test functions :
    • totally separates test code, test configuration and test setup
    • ideal for integration and functional tests
    • allows for flexible and natural test parametrization schemes
  • new plugin architecture, allowing easy-to-write project-specific and cross-project single-file plugins. The most notable new external plugin is oejskit which naturally enables running and reporting of javascript-unittests in real-life browsers.
  • many new features done in easy-to-improve default plugins, highlights:
    • xfail: mark tests as "expected to fail" and report separately.
    • pastebin: automatically send tracebacks to pocoo paste service
    • capture: flexibly capture stdout/stderr of subprocesses, per-test …
    • monkeypatch: safely monkeypatch modules/classes from within tests
    • unittest: run and integrate traditional unittest.py tests
    • figleaf: generate html coverage reports with the figleaf module
    • resultlog: generate buildbot-friendly reporting output
  • distributed testing and elastic distributed execution:
    • new unified "TX" URL scheme for specifying remote processes
    • new distribution modes "–dist=each" and "–dist=load"
    • new sync/async ways to handle 1:N communication
    • improved documentation

The py lib continues to offer most of the functionality used by the testing tool in independent namespaces.

Some non-test related code, notably greenlets/co-routines and api-generation now live as their own projects which simplifies the installation procedure because no C-Extensions are required anymore.

The whole package should work well with Linux, Win32 and OSX, on Python 2.3, 2.4, 2.5 and 2.6. (Expect Python3 compatibility soon!)

For more info, see the py.test and py lib documentation:

http://pytest.org

http://pylib.org

have fun, holger

Written by holger krekel

August 4, 2009 at 10:05 am

py.test: shrinks code, grows plugins and integration testing

leave a comment »

Just before Pycon i uploaded the lean and mean py.test 1.0.0b1 beta release. A lot of code got moved out, most notably the greenlets C-extension. This simplifies packaging and increases the py lib’s focus on test facilities. It now has a pluginized architecture and provides funcargs which tremendously help with writing functional and integration tests. One such example are py.test’s own acceptance tests checking behaviour of the command line tool from a user perspective. Other features include a zero-install mechanism for distributing tests which also allow to conveniently drive cross-platform integration tests.

Unittesting, functional and integration testing are now official targets. No doubt, Test category naming is a slippery subject and it’s a good idea to consider test category names as labels rather than "either-or" categories. In the end, tests are about being useful for software development which today means coding for a wide variety of environments and involving integration and deployment issues on every corner. I think that testing tools yet have to develop their full potential. In my opinion, automated testing and deployment techniques are to fully integrate with each other, and i consider coding of distributed integration test scenarios as key to that.

The upcoming final py.test 1.0 release i’d like to make a starting point for facilitating the integration of many more test methods and test mechanisms via plugins. Some people already contributed a pytest_figleaf (for coverage testing) and pytest_twisted (for running twisted style tests) although i am still only finalizing API details and writing up docs. So I am very happy how things are turning out and also motivated by the positive feedback on the two testing tutorials that Brian Dorsey and me gave at Pycon (see his writeup).

Btw, if you use the quickstart and encounter any problems, please use the brand new issue tracker on bitbucket. I started hosting a mercurial py.test trunk repository and so far it’s been a positive experience and I guess I fully switch to mercurial soon. Alas, i probably drop setuptools before i go 1.0 with py.test – it simply causes too many troubles. py.test’s trunk has a straightforward setup.py and i intend to release a second beta with refined docs and removed setuptools. Stay tuned for many more news in May – right now, i am looking forward to some 1-week offline holiday :)

cheers, holger

Written by holger krekel

April 18, 2009 at 9:05 pm

Mobiles, Python, PyPy and the Zone VM

with 2 comments

Next week i am going to OpenBossa, a developer conference organised by Nokia’s research institute INDT in Brazil. It’s about free and open source developments on small internet devices. I’ll be talking about PyPy on Maemo. In a nutshell, last year we made PyPy cross-compile for the Maemo/Linux platform. It turned out to use less RAM than CPython for virtually all Python objects. It also starts new Python processes faster. We have ideas to make it feasible to run hundreds of small isolated python processes on machines like my 190 Euro mobile or even cheaper ones. I’ll post my slides with some more details next week and probably also twitter and blog a bit about my conference experiences.

To be honest, I’ve long ignored developments in the mobile device world. Maybe mostly because I want to interact with people through open and freely available networks. The way in which mobile networks are run for commercial purposes hampered my use of mobiles and my general interest.

But things seem to be changing rapidly. I appreciate that. There are more and more phones who connect both to mobile networks and to WLANs. The sudden possibility to use the internet whereever i have WLAN access opens up the device for me. My small mobile has 128 MByte and a 250 MHZ ARM CPU. It runs Python. I can freely download tools from other people and use them. Or I can hack something up or get in contact with developers. I can use EMail, Internet radio, Twitter or browse the internet to begin with. I find the software generally likeable although a bit shaky to install and control sometimes.

Well, I’d like to change various aspects of the User interface but they are probably quite hard-wired, e.g. text completion. Nevertheless, i am thrilled by the overall technical opportunities. This device could soon operate my working environment (it has a 16 Gig SD-RAM card for 15 Euros) and become a center piece of collaboration and communication. For this i’d like to trust the software and the people behind it. So I look forward to meet up with developers more familiar with mobile worlds. And to hang around Porto de Galinhas which i have been told is a beautiful place.

Of course, I am also curious to present and discuss how PyPy, with some more efforts, could provide a perfectly fitting Pythonic environment for mobiles. My vision is to have a Python environment for driving all technical aspects of the phone and to instantly share scripts and apps with fellows worldwide. All on the basis of a decentral network comprised of devices participating in it. Similar to what Charles Stross describes in his books as the "Zone VM": a distributed virtual machine and environment.

cheers, holger

Written by holger krekel

March 4, 2009 at 8:35 pm

Posted in metaprogramming

Tagged with , ,

Monkeypatching in unit tests, done right

with 13 comments

[updated, thanks to marius].

I am currently preparing my testing tutorials for Pycon and here is an example i’d lke to share already.

The problem: In a test function we want to patch an Environment variable and see if our application handles something related to it correctly. The direct approach for doing this in a test function might look like this:

def test_envreading(self):
    old = os.environ['ENV1']
    os.environ['ENV1'] = 'myval'
    try:
        val = myapp().readenv()
        assert val == "myval"
    finally:
        os.environ['ENV1'] = old

If we needed to do this several times for test functions we’d have a lot of repetetive boilerplatish code. The try-finally and undo-related code does not even take into account that ENV1 might not have been set originally.

Most experienced people would use setup/teardown methods to get less-repetetive testing code. We might end up with something slightly more general like this:

def setup_method(self, method):
    self._oldenv = os.environ.copy()

def teardown_method(self, method):
    os.environ.update(self._oldenv)

def test_envreading(self):
    os.environ['ENV1'] = "myval"
    val = myapp().readenv()
    assert val == "myval"

This avoids repetition of setup code but it scatters what belongs to the test function across three functions. All other functions in the Testcase class will get the service of a preserved environment although they might not need it. If i want to move away this testing function i will need to take care to copy the setup code as well. Or i start subclassing Test cases to share code. If we then start to need modifying other dicts or classes we have to add code in three places.

Monkeypatching the right way

Here is a version of the test function which uses pytest’s monkeypatch` plugin. The plugin does one thing: it provides a monkeypatch object for each test function that needs it. The resulting test function code then looks like this:

def test_envreading(self, monkeypatch):
    monkeypatch.setitem(os.environ, 'ENV1', 'myval')
    val = myapp().readenv()
    assert val == "myval"

Here monkeypatch.setitem() will memorize old settings and modify the environment. When the test function finishes the monkeypatch object restores the original setting. This test function is free to get moved across files. No other test function or code place is affected or required to change when it moves.

Let’s take a quick look at the "providing" side, i.e. the pytest_monkeypatch.py plugin which provides "Monkeypatch" instances to test functions. It makes use of pytest’s new pyfuncarg protocol.

The plugin itself is free to get refined and changed as well, without affecting the existing test code. The following 71 lines of code make up the plugin, including tests:

class MonkeypatchPlugin:
    """ setattr-monkeypatching with automatical reversal after test. """
    def pytest_pyfuncarg_monkeypatch(self, pyfuncitem):
        monkeypatch = MonkeyPatch()
        pyfuncitem.addfinalizer(monkeypatch.finalize)
        return monkeypatch

notset = object()

class MonkeyPatch:
    def __init__(self):
        self._setattr = []
        self._setitem = []

    def setattr(self, obj, name, value):
        self._setattr.insert(0, (obj, name, getattr(obj, name, notset)))
        setattr(obj, name, value)

    def setitem(self, dictionary, name, value):
        self._setitem.insert(0, (dictionary, name, dictionary.get(name, notset)))
        dictionary[name] = value

    def finalize(self):
        for obj, name, value in self._setattr:
            if value is not notset:
                setattr(obj, name, value)
            else:
                delattr(obj, name)
        for dictionary, name, value in self._setitem:
            if value is notset:
                del dictionary[name]
            else:
                dictionary[name] = value


def test_setattr():
    class A:
        x = 1
    monkeypatch = MonkeyPatch()
    monkeypatch.setattr(A, 'x', 2)
    assert A.x == 2
    monkeypatch.setattr(A, 'x', 3)
    assert A.x == 3
    monkeypatch.finalize()
    assert A.x == 1

    monkeypatch.setattr(A, 'y', 3)
    assert A.y == 3
    monkeypatch.finalize()
    assert not hasattr(A, 'y')


def test_setitem():
    d = {'x': 1}
    monkeypatch = MonkeyPatch()
    monkeypatch.setitem(d, 'x', 2)
    monkeypatch.setitem(d, 'y', 1700)
    assert d['x'] == 2
    assert d['y'] == 1700
    monkeypatch.setitem(d, 'x', 3)
    assert d['x'] == 3
    monkeypatch.finalize()
    assert d['x'] == 1
    assert 'y' not in d

def test_monkeypatch_plugin(testdir):
    sorter = testdir.inline_runsource("""
        pytest_plugins = 'pytest_monkeypatch',
        def test_method(monkeypatch):
            assert monkeypatch.__class__.__name__ == "MonkeyPatch"
    """)
    res = sorter.countoutcomes()
    assert tuple(res) == (1, 0, 0), res

I can also imagine some nice plugin which supports mock objects – patching methods with some preset behaviour or tracing calls between components.

have fun, holger

Written by holger krekel

March 3, 2009 at 1:48 pm

Follow

Get every new post delivered to your Inbox.