metaprogramming and politics

Decentralize. Take the red pill.

metaprogramming in Python: What CPython, PyPy, Pyramid, pytest and politics have in common …

leave a comment »

Metaprogramming in Python too often revolves around metaclasses, which are just a narrow application of the “meta” idea and not a great one at that. Metaprogramming more generally deals with reasoning about program code, about taking a “meta” stance on it.  A metaprogram takes a program as input, often just partial programs like functions or classes. Here are a few applications of metaprogramming:

  • CPython is a metaprogram written in C. It takes Python program code as input and interprets it, so that it runs at a higher level than C.
  • PyPy is a metaprogramm written in Python. It takes RPython program code as input and generates a C-level metaprogram (the PyPy interpreter) which itself interprets Python programs and takes another meta stance by generating Assembler pieces for parts of the interpreation execution. If you like, PyPy is a metaprogram generating metaprograms whereas CPython and typical compilers like GCC are “just” a metaprogram.
  • Pyramid is a metaprogram that takes view, model definitions and http-handling code as input and executes them, thereby raising code on a higher level to implement the “Pyramid application” language.
  • pytest is a metaprogram written in Python, taking test, fixture and plugin functions as input and executing them in a certain manner, thereby implementing a testing language.
  • metaclasses: in Python they allow to intercept class creation and introspect methods and attributes, amending their behaviour. Because metaclass-code usually executes at import time, it often uses global state for implementing non-trivial meta aspects.

Apart from these concrete examples, language compilers, testing tools and web frameworks all have metaprogramming aspects. Creating big or small “higher” level or domain-specific languages within Python is as a typical example of metaprogramming. Python is actually a great language for metaprogramming although it could be better.

In future blog posts i plan to talk about some good metaprogramming practise, particularly:

  • keep the layers/levels separate by good naming and API design
  • define a concise “language” for the programs you take as input
  • avoid creating global state in your metaprograms (and elsewhere)
    which can easily happen with meta-classes executing at import time

Lastly, i see metaprogramming at work not only when coding in a computer language. Discussing the legal framing for executing programs on the internet is some kind of metaprogramming, especially if you consider licensing and laws as human-interpreted code which affects how programs can be written, constructed and executed. In reverse, web applications increasingly affect how we interact with each other other, thereby implementing rules formerly dealt with in the arena of politics. Therefore, metaprogramming and politics are fundamentally connected topics.

have metafun, i. e. take fun stuff as input to generate more of it :) holger

Written by holger krekel

November 22, 2012 at 3:04 pm

execution locals: better than thread locals/globals

with 6 comments

While many agree that global state is evil, the so called “thread locals” are not much better. Even though they help to separate state on a per-thread or per-greenlet basis, they still are global within that context. In particular (thread) global state means that:

  • Invoked functions can change bindings of an invoking function as a side effect
  • thread locals may linger around even if their state is not used or became invalid

Meet “execution locals” which avoid these problems. Find the code released on PyPI:

http://pypi.python.org/pypi/xlocal

It’s some 60 lines of code and tested on python2.5 up to python3.3 and pypy and ready to be played with. I inline its README.txt below in case you can’t or don’t want to switch reading context. One more note: If I were to design a new language i’d probably remove “globals” all together and only offer something like the “xlocal” type with a more straight forward syntax.

execution locals: killing global state (including thread locals)

The xlocal module provides execution locals aka “xlocal” objects which implement a more restricted variant of “thread locals”. An “xlocal” instance allows to manage its attributes on a per-execution basis in a manner similar to how real locals work:

  • Invoked functions cannot change the binding for the invoking function
  • existence of a binding is local to a code block (and everything it calls)

Attribute bindings for an xlocal object will not leak outside a context-managed code block and they will not leak to other threads or greenlets. By contrast, both process-globals and “thread locals” do not implement these properties.

Let’s look at a basic example:

# content of example.py

from xlocal import xlocal

xcurrent = xlocal()

def output():
    print "hello world", xcurrent.x

if __name__ == "__main__":
    with xcurrent(x=1):
        output()

If we execute this module, the output() function will see a xcurrent.x==1 binding:

$ python example.py
hello world 1

Here is what happens in detail: xcurrent(x=1) returns a context manager which sets/resets the x attribute on the xcurrent object. While remaining in the same thread/greenlet, all code triggered by the with-body (in this case just the output() function) can access xcurrent.x. Outside the with- body xcurrent.x would raise an AttributeError. It is also not allowed to directly set xcurrent attributes; you always have to explicitely mark their life-cycle with a with-statement. This means that invoked code:

  • cannot rebind xlocal state of its invoking functions (no side effects, yay!)
  • xlocal state does not leak outside the with-context (lifecylcle control)

Another module may now reuse the example code:

# content of example_call.py
import example

with example.xcurrent(x=3):
    example.output()

which when running …:

$ python example_call.py
hello world 3

will cause the example.output() function to print the xcurrent.x binding as defined at the invoking with xcurrent(x=3) statement.

Other threads or greenlets will never see this xcurrent.x binding; they may even set and read their own distincit xcurrent.x object. This means that all threads/greenlets can concurrently call into a function which will always see the execution specific x attribute.

Usage in frameworks and libraries invoking “handlers”

When invoking plugin code or handler code to perform work, you may not want to pass around all state that might ever be needed. Instead of using a global or thread local you can safely pass around such state in execution locals. Here is a pseudo example:

xcurrent = xlocal()

def with_xlocal(func, **kwargs):
    with xcurrent(**kwargs):
        func()

def handle_request(request):
    func = gethandler(request)  # some user code
    spawn(with_xlocal(func, request=request))

handle_request will run a user-provided handler function in a newly spawned execution unit (for example spawn might map to threading.Thread(…).start() or to gevent.spawn(…)). The generic with_xlocal helper wraps the execution of the handler function so that it will see a xcurrent.request binding. Multiple spawns may execute concurrently and xcurrent.request will carry the execution-specific request object in each of them.

Issues worth noting

If a method decides to memorize an attribute of an execution local, for example the above xcurrent.request, then it will keep a reference to the exact request object, not the per-execution one. If you want to keep a per-execution local, you can do it this way for example:

Class Renderer:
    @property
    def request(self):
        return xcurrent.request

this means that Renderer instances will have an execution-local self.request object even if the life-cycle of the instance crosses execution units.

Another issue is that if you spawn new execution units, they will not implicitely inherit execution locals. Instead you have to wrap your spawning function to explicitely set execution locals, similar to what we did in the above “invoking handlers” section.

Written by holger krekel

November 16, 2012 at 2:22 pm

If i were to design a new programming language …

with 18 comments

I’d see to base syntax and semantics on Python3, but strip and rebase it:

  • no C: implement the interpreter in RPython, get a JIT for free and implementation bits from PyPy’s Python interpreter (parsing, IO, etc.)
  • no drags-you-down batteries: lean interpreter core and a standard battery distro which is tested against the last N interpreter versions + current
  • no yield: use greenlets to implement all of what yield provides and more
  • no underlying blocking on IO: base it all on event loop, yet provide synchronous programming model through greenlets
  • no c-level API nor ctypes: use cffi to interface with c-libraries
  • no global state: just support state bound to execution context/stack
  • no GIL: support free threading and Automatic Mutual Exclusion for dealing with shared state
  • no setup.py: have a thought-through story and tools from the start for packaging, installation, depending/interfacing between packages
  • no import, no sys.modules: provide an object with which you can access other packages’s objects and introspect/interact with one’s own package
  • no testing as an afterthought: everything needs to be easily testable, empowered assert statement and branch-coverage supported from the core.
  • no extensibility as an afterthought: support plugins and loose coupling through builtin 1:N calling mechanism (event notification on steroids)
  • no unsafe code: support IO/CPU/RAM sandboxing as a core feature
  • no NIH syndrome: provide a bridge to a virtualenv’ed Python interpreter allowing to leverage existing good crap

Anything else? Probably! Discussion needed? Certainly. Unrealistic? Depends on who would participate — almost all of the above has projects, PEPs and code showcasing viability.

Btw, did you know that when we started PyPy we initially did this under the heading of “Minimal Python”? Some of the above ideas above and their underlying motivations were already mentioned when I invited to the first PyPy sprint almost 10 years ago:

http://mail.python.org/pipermail/python-dev/2003-January/032427.html

I learned since then that Python has more complex innards than it seems but i still believe it could be both simpler and more powerful.

holger

Written by holger krekel

November 13, 2012 at 3:29 pm

Why privacy matters and State Secrecy not

Today i saw a woman’s german article on how she wouldn’t want reports about her sex life spread through wikileaks. I wouldn’t like that either. But i see it as a misconception and confusion about two very different issues, namely personal privacy and state/government secrecy. This confusion is used and increased by many politicians for their own purposes.

Privacy means our rights to have a private life, private flat, private actions and private communications. We don’t want the state or the public to surveil us or intrude our private world unless they can show evidence to an independent court that there is something criminal going on.

State secrecy denotes making secret deals, performing secret communication with and secret actions against people or other states. Secret actions can naturally not be discussed in the public and are exempt from our judgement when electing officials. Increasing state secrecy very quickly leads to inner circles wielding great power. The 20th century has tons of bad examples.

Therefore I refuse the notion that if am positive about privacy i must also be ok with state secrecy. Or if i want a transparent government that i also must be ok with total surveillance of my private life. No way. Likely it’s rather true that the more transparent a government is the more secure i can feel with respect to my privacy.

On a sidenote, this all relates to a point in the hacker ethics made by Wau Holland and the Chaos Computer Club a long time ago: “Make public data available, protect private data.”

Written by holger krekel

December 5, 2010 at 7:34 pm

Posted in politics

Tagged with

Wikileaks or: Welcome to the brave new “Terms of Use” era

with 8 comments

Paypal “restricts” the bank account of the german Wau Holland foundation which managed parts of Wikileaks transactions. Citing from their statement :

PayPal has permanently restricted the account used by WikiLeaks due to
a violation of the PayPal Acceptable Use Policy, which states that
our payment service cannot be used for any activities that
encourage, promote, facilitate or instruct others to engage in
illegal activity. We’ve notified the account holder of this action.

As with my yesterday notes a number of questions arise. What exactly is
illegal about wikileaks behaviour in the US? Was money laundering or
online fraud involved? Was wikileaks given a warning and asked to
provide clarifying statements regarding the usage of _their_ money? Do
they apply the same moral standards to all their customers and accounts?
For example, are they also closing all accounts related to the
Washington Times or related to Sarah Palin publically calling for
assasination
of a citizen of Australia?

If we are learning anything from the Wikileaks case it is the willfullness
of young IT companies in the US to concur with public opinion or direct
government pressure. Up until now i presumed the likes of Amazon or Paypal
also incorporated some ethics. And this would mandate IMO to keep
distance to government actions and to resist government pressure especially
if it’s about something like wikileaks. It seems like the US goverment
used its advanced notice of the pending publication wisely: rather than filtering critical
messages they organised a huge campaign in the US resulting in Amazon,
PayPal and everydns and probably others to implement government policies
and quickly. No need to have special laws or evil government actions – just reference the Terms of Use and be done.

Written by holger krekel

December 4, 2010 at 3:11 pm

Posted in politics

Tagged with

Who needs censorship if you have Amazon and everydns?

with 13 comments

Yesterday Amazon stopped services for the wikileaks archive. Citing from the Amazon statement:

It is not credible that the extraordinary volume of 250,000 classified documents that WikiLeaks is publishing could have been carefully redacted in such a way as to ensure that they weren’t putting innocent people in jeopardy.

So here you have a popular cloud services provider judging their customers content in a broad manner, stating conclusion, terminate services, done. Did they allow discussion? Do journalists need to keep and control all rights of the material they are publishing? Can Amazon show any evidenceof their “jeopardy” conclusion? Would they apply the same moral standard to e.g. blogs or other (journalistic) content that called for going to the war on Iraq – causing >100.000 civilians to die? Does Amazon really want us to believe the US governments outrage and Mr. Liebermanns actions are merely a co-incidence?

Then today everydns.net terminated DNS services for wikileaks.org with a 24 hour prior notice. Citing from the everydns statement:

More specifically, the services were terminated for violation of the provision which states that “Member shall not interfere with another Member’s use and enjoyment of the Service or another entity’s use and enjoyment of similar services.”

So supposedly there was an attack on their DNS servers and they interpreted this as wikileaks doing harm to others? Is this the future of how we handle attacks against single domain names? Would the same happen if it was the site of a chinese dissident or a Poker playing site or any other customer?

As things stand this puts me off these two companies and brings me again to the thought that we need an internet that is as independent from any single company or any single country as possible - it’s in the best interest for all of us in the long run.

Written by holger krekel

December 3, 2010 at 12:44 pm

Posted in politics

Tagged with

Ring of Python talk at pycon

with 3 comments

Just did my talk on Ring of Python talk at Pycon US 2010, discussing competition and features of Python interpreters and co-operation issues around the most important issue in my oppinion: deployment. Also showcased execnet as a generic Python2Python bridge, connecting Python2.4, Python 3.1, Jython and IronPython. Got some nice feedback, also about the presentation style, actually i was using Prezi. You may go to the following page, click into the flash app and hit “cursor right”: Ring of Python .

Written by holger krekel

February 19, 2010 at 11:06 pm

Posted in metaprogramming

Follow

Get every new post delivered to your Inbox.