Archive for November 2012
Metaprogramming in Python too often revolves around metaclasses, which are just a narrow application of the “meta” idea and not a great one at that. Metaprogramming more generally deals with reasoning about program code, about taking a “meta” stance on it. A metaprogram takes a program as input, often just partial programs like functions or classes. Here are a few applications of metaprogramming:
- CPython is a metaprogram written in C. It takes Python program code as input and interprets it, so that it runs at a higher level than C.
- PyPy is a metaprogramm written in Python. It takes RPython program code as input and generates a C-level metaprogram (the PyPy interpreter) which itself interprets Python programs and takes another meta stance by generating Assembler pieces for parts of the interpreation execution. If you like, PyPy is a metaprogram generating metaprograms whereas CPython and typical compilers like GCC are “just” a metaprogram.
- Pyramid is a metaprogram that takes view, model definitions and http-handling code as input and executes them, thereby raising code on a higher level to implement the “Pyramid application” language.
- pytest is a metaprogram written in Python, taking test, fixture and plugin functions as input and executing them in a certain manner, thereby implementing a testing language.
- metaclasses: in Python they allow to intercept class creation and introspect methods and attributes, amending their behaviour. Because metaclass-code usually executes at import time, it often uses global state for implementing non-trivial meta aspects.
Apart from these concrete examples, language compilers, testing tools and web frameworks all have metaprogramming aspects. Creating big or small “higher” level or domain-specific languages within Python is as a typical example of metaprogramming. Python is actually a great language for metaprogramming although it could be better.
In future blog posts i plan to talk about some good metaprogramming practise, particularly:
- keep the layers/levels separate by good naming and API design
- define a concise “language” for the programs you take as input
- avoid creating global state in your metaprograms (and elsewhere)
which can easily happen with meta-classes executing at import time
Lastly, i see metaprogramming at work not only when coding in a computer language. Discussing the legal framing for executing programs on the internet is some kind of metaprogramming, especially if you consider licensing and laws as human-interpreted code which affects how programs can be written, constructed and executed. In reverse, web applications increasingly affect how we interact with each other other, thereby implementing rules formerly dealt with in the arena of politics. Therefore, metaprogramming and politics are fundamentally connected topics.
have metafun, i. e. take fun stuff as input to generate more of it :) holger
While many agree that global state is evil, the so called “thread locals” are not much better. Even though they help to separate state on a per-thread or per-greenlet basis, they still are global within that context. In particular (thread) global state means that:
- Invoked functions can change bindings of an invoking function as a side effect
- thread locals may linger around even if their state is not used or became invalid
Meet “execution locals” which avoid these problems. Find the code released on PyPI:
It’s some 60 lines of code and tested on python2.5 up to python3.3 and pypy and ready to be played with. I inline its README.txt below in case you can’t or don’t want to switch reading context. One more note: If I were to design a new language i’d probably remove “globals” all together and only offer something like the “xlocal” type with a more straight forward syntax.
execution locals: killing global state (including thread locals)
The xlocal module provides execution locals aka “xlocal” objects which implement a more restricted variant of “thread locals”. An “xlocal” instance allows to manage its attributes on a per-execution basis in a manner similar to how real locals work:
- Invoked functions cannot change the binding for the invoking function
- existence of a binding is local to a code block (and everything it calls)
Attribute bindings for an xlocal object will not leak outside a context-managed code block and they will not leak to other threads or greenlets. By contrast, both process-globals and “thread locals” do not implement these properties.
Let’s look at a basic example:
# content of example.py from xlocal import xlocal xcurrent = xlocal() def output(): print "hello world", xcurrent.x if __name__ == "__main__": with xcurrent(x=1): output()
If we execute this module, the output() function will see a xcurrent.x==1 binding:
$ python example.py hello world 1
Here is what happens in detail: xcurrent(x=1) returns a context manager which sets/resets the x attribute on the xcurrent object. While remaining in the same thread/greenlet, all code triggered by the with-body (in this case just the output() function) can access xcurrent.x. Outside the with- body xcurrent.x would raise an AttributeError. It is also not allowed to directly set xcurrent attributes; you always have to explicitely mark their life-cycle with a with-statement. This means that invoked code:
- cannot rebind xlocal state of its invoking functions (no side effects, yay!)
- xlocal state does not leak outside the with-context (lifecylcle control)
Another module may now reuse the example code:
# content of example_call.py import example with example.xcurrent(x=3): example.output()
which when running …:
$ python example_call.py hello world 3
will cause the example.output() function to print the xcurrent.x binding as defined at the invoking with xcurrent(x=3) statement.
Other threads or greenlets will never see this xcurrent.x binding; they may even set and read their own distincit xcurrent.x object. This means that all threads/greenlets can concurrently call into a function which will always see the execution specific x attribute.
Usage in frameworks and libraries invoking “handlers”
When invoking plugin code or handler code to perform work, you may not want to pass around all state that might ever be needed. Instead of using a global or thread local you can safely pass around such state in execution locals. Here is a pseudo example:
xcurrent = xlocal() def with_xlocal(func, **kwargs): with xcurrent(**kwargs): func() def handle_request(request): func = gethandler(request) # some user code spawn(with_xlocal(func, request=request))
handle_request will run a user-provided handler function in a newly spawned execution unit (for example spawn might map to threading.Thread(…).start() or to gevent.spawn(…)). The generic with_xlocal helper wraps the execution of the handler function so that it will see a xcurrent.request binding. Multiple spawns may execute concurrently and xcurrent.request will carry the execution-specific request object in each of them.
Issues worth noting
If a method decides to memorize an attribute of an execution local, for example the above xcurrent.request, then it will keep a reference to the exact request object, not the per-execution one. If you want to keep a per-execution local, you can do it this way for example:
Class Renderer: @property def request(self): return xcurrent.request
this means that Renderer instances will have an execution-local self.request object even if the life-cycle of the instance crosses execution units.
Another issue is that if you spawn new execution units, they will not implicitely inherit execution locals. Instead you have to wrap your spawning function to explicitely set execution locals, similar to what we did in the above “invoking handlers” section.
Copyright / inspiration
This code is based on discussions with Armin Ronacher and others in response to a tweet of mine. It extracts and refines some ideas found in Armin’s “werzeug.local” module and friends.
MIT, see LICENSE for more details.
With this “xlocal” experiment i am trying to explore how much of new language ideas can be prototyped with existing Python. For more related goals, see my earlier post on If i were to design a new language.
I’d see to base syntax and semantics on Python3, but strip and rebase it:
- no C: implement the interpreter in RPython, get a JIT for free and implementation bits from PyPy’s Python interpreter (parsing, IO, etc.)
- no drags-you-down batteries: lean interpreter core and a standard battery distro which is tested against the last N interpreter versions + current
- no yield: use greenlets to implement all of what yield provides and more
- no underlying blocking on IO: base it all on event loop, yet provide synchronous programming model through greenlets
- no c-level API nor ctypes: use cffi to interface with c-libraries
- no global state: just support state bound to execution context/stack
- no GIL: support free threading and Automatic Mutual Exclusion for dealing with shared state
- no setup.py: have a thought-through story and tools from the start for packaging, installation, depending/interfacing between packages
- no import, no sys.modules: provide an object with which you can access other packages’s objects and introspect/interact with one’s own package
- no testing as an afterthought: everything needs to be easily testable, empowered assert statement and branch-coverage supported from the core.
- no extensibility as an afterthought: support plugins and loose coupling through builtin 1:N calling mechanism (event notification on steroids)
- no unsafe code: support IO/CPU/RAM sandboxing as a core feature
- no NIH syndrome: provide a bridge to a virtualenv’ed Python interpreter allowing to leverage existing good crap
Anything else? Probably! Discussion needed? Certainly. Unrealistic? Depends on who would participate — almost all of the above has projects, PEPs and code showcasing viability.
Btw, did you know that when we started PyPy we initially did this under the heading of “Minimal Python”? Some of the above ideas above and their underlying motivations were already mentioned when I invited to the first PyPy sprint almost 10 years ago:
I learned since then that Python has more complex innards than it seems but i still believe it could be both simpler and more powerful.