My “speed up pypi installs” PEP438 has been accepted and transition phase 1 is live: as a package maintainer you can speed up the installation for your packages for all your users now, with the click of a button: Login to
and then go to urls for each of your packages, and specify that all release files are hosted from pypi.python.org. Or add explicit download urls with an MD5. Tools such as pip or easy_install will thus avoid any slow crawling of third party sites.
Many thanks to Carl Meyer who helped me write the PEP, and Donald Stufft for implementing most of it, and Richard Jones who accepted it today! And thanks also to the distutils-sig discussion participants, in particular Phillip Eby and Marc-Andre Lemburg.
If a man were to tweet a mysogynist joke and his followers were men, would that be an issue? What if one of them re-tweets it and one of his female followers complains on twitter? And then many other people start tweeting and re-tweeting this or that and what if this all got the initial tweeter fired from his company? And then her company would fire her as well?
Quite a mess, obviously. However, I think everyone had their reasons for talking and acting the way they did. And it boils down to the perspective you are able to feel empathy for. Here is a possible set of perspectives:
Perspective M: “The other day i was ridiculed by a bunch of girls at the office. I wanted to pay back with a little joke in an environment where i felt safe to do so.”
Perspective F: “Again a tweet with bad mysogynist jokes. I’ve had enough. This time i won’t sit quietly but call it out.”
Perspective C1: “Damn, look what this guy caused. His twitter profile is directly associated with our company. And now he tells bad mysogynist jokes and it’s now all over the internet. We cannot let go this time.”
Perspective C2: “Damn it, look what she caused. She is working in public relations and doesn’t know better than to cause a shitstorm which directly comes back to us a company? We cannot let go.”
I could understand each of these perspectives though i’d have a suspicision that the companies choose a bit of an easy way out. Had they rather put the issue of misogyny at the center of their positioning and communication, rather than focusing just on keeping some damage from the company, everybody would have learned a lesson and the incident could have contributed to a more enjoyable environment, i am sure.
Besides the good feedback and discussions around my talk, i just had a great few days. It was my first time to Russia and i saw and learned a lot. One unexpected event was going to a russian Sauna with Amir Salihefendic, Russel Keith-Magee and Anton, a main conference organizer. Between going into the Sauna we had glasses of nice irish Whiskey or walked outside to the snowy freezing cold. Afterwards some of us went to the conference party and had good (despite being somewhat drunken) discussions with people from Yandex, the biggest russian search engine and several russian devs. All very friendly, competent and funny. The party lasted until 5:30am – with my fellow english-speaking talkers Armin Ronacher, David Cramer (a weekend in Russia) and me being among the very last.
The next days evening saw Amir, David, Armin and two russian guys visiting an Irish pub past midnight. It turned out there is no such thing as a “russian pub”, the concept of “pub” was imported in the last decade mostly in the form of english or irish ones. And it seems IT/Python guys can meet everywhere on the planet and have a good time
Going back to content, i felt particularly inspired by Jeff Lindsay’s talk on Autosustainable services. He described how he tries to provide several small web services, and how to organize cost sharing by its users. As services need resources, it’s a different issue than Open Cource collaboration which does not require such to exist.
I heart several good sentences from my fellow talkers, for example one from Russel Keith-Magee describing a dillema of open source communities: “There are many people who can say ‘No’ but few who can say ‘Yes’ to something”. Amir Salihefendic desribed how the “Redis” database solved many problems for him, and some interesting concrete usages of “bitmaps” in his current endeavours like bitmapist.cohort. And of course Armin Ronacher and David Cramer also gave good talks related to their experience, Advanced Flask patterns and scalable web services respectively. With Armin i also had a good private discussion about the issue of code-signing and verification. We drafted what we think could work for Python packaging (more separately). With David, i discussed workflow commands for python packaging as he offered some good thoughts on the matter.
Around the whole conference we were warmly cared for by Yulia’s company it-people.ru who overtook the physical organisation, and by Anton and his friends who organized the program. Maria Kalinina in particular had cared for the keynote speakers and many other aspects of the conference, and without her, i wouldn’t have made it. Anton drove us to the Asian European geographic border, and Yulia to the skyscraper of Ekaterinburg, overlooking the third largest city in Russia. Russel and me also took the opportunity to walk around Ekaterinburg, looking at Lenin sculptures, buildings made of ice, frozen lakes, and the many shops and noises in the city.
Lastly i went to the university with Russel to talk for two hours to students about “How Open Source can help your career” and we had a lively discussion with them and the lecturer who invited us. I offered my own background and stated that the very best people in the IT world are today collaborating through open-source. It’s a totally dominant model for excellence. (Which doesn’t mean there are not some good proprietary projects, they are just fewer i’d say).
So i can join the many russian participants who thought Pycon Russia was a very good conference. It’s of course mostly interesting for people speaking russian, as only seven talks were in english. For my part, the intense time i had with both the russian hosts and developers and the english talkers was verymuch worth it – i think there might be a few new collaborations coming from it. More on that in later blog posts hopefully
Two days ago i left Ekaterinburg and felt a bit sad because of the many contacts i made, which almost felt like the beginning of friendships.
PSF’s code of conduct enforcement is a good step, but what about the many traditional family models in the IT world? I know many fathers which are busy fulltime with non-child stuff, and their partners have the main child responsibility. I heart three main reasonings for this situation and i don’t fully buy them:
- an economic one: the guy working brings more money into the household. This kind of perpetuates the inequality situation, doesn’t it? And is having less money really an issue? Is part-time working impossible? In germany you have a legal right to do part-time work, to begin with.
- a biologistic one: women can “naturally” or genetically care better than men for children. One, I’ve seen fathers doing just fine. Two, are we entirely determined by genetics? I see genetics as some kind of hardware, and software can do lots of different things on it. Culture is shaped as much as software. There is no such thing as “objective” nature.
- go away, it’s a family’s private business and choices. Nevertheless such choices are also culturally determined. Often there is no explicit discussion or choice but rather a fallback to the default, often induced by the facts of birth and breast feeding. How many fathers discuss the issue of child-care openly and regularly, offering changes to give a real choice?
Rest assured, I really like the projects i am hacking on as much as the other guy. Sometimes i feel that caring often for my child makes this harder. On the plus side, it gives me better focus because my time is more limited. And more often than not, i am grateful and have a lot of fun being with my little one.
Now, if more fathers in the Python communities were busier with their children, what would that change in terms of conference attendance of women? Not sure there would be any direct effect except maybe lower conference attendance of men, rising the percentage of women. It would set a good example, however, and help mid- to long-term, i am sure.
Sometimes i like to ask myself this question: when i am dying and wonder what should i have done rather differently? I doubt i am going to say “i should have released one more library, earned more money, become more popular”.
Metaprogramming in Python too often revolves around metaclasses, which are just a narrow application of the “meta” idea and not a great one at that. Metaprogramming more generally deals with reasoning about program code, about taking a “meta” stance on it. A metaprogram takes a program as input, often just partial programs like functions or classes. Here are a few applications of metaprogramming:
- CPython is a metaprogram written in C. It takes Python program code as input and interprets it, so that it runs at a higher level than C.
- PyPy is a metaprogramm written in Python. It takes RPython program code as input and generates a C-level metaprogram (the PyPy interpreter) which itself interprets Python programs and takes another meta stance by generating Assembler pieces for parts of the interpreation execution. If you like, PyPy is a metaprogram generating metaprograms whereas CPython and typical compilers like GCC are “just” a metaprogram.
- Pyramid is a metaprogram that takes view, model definitions and http-handling code as input and executes them, thereby raising code on a higher level to implement the “Pyramid application” language.
- pytest is a metaprogram written in Python, taking test, fixture and plugin functions as input and executing them in a certain manner, thereby implementing a testing language.
- metaclasses: in Python they allow to intercept class creation and introspect methods and attributes, amending their behaviour. Because metaclass-code usually executes at import time, it often uses global state for implementing non-trivial meta aspects.
Apart from these concrete examples, language compilers, testing tools and web frameworks all have metaprogramming aspects. Creating big or small “higher” level or domain-specific languages within Python is as a typical example of metaprogramming. Python is actually a great language for metaprogramming although it could be better.
In future blog posts i plan to talk about some good metaprogramming practise, particularly:
- keep the layers/levels separate by good naming and API design
- define a concise “language” for the programs you take as input
- avoid creating global state in your metaprograms (and elsewhere)
which can easily happen with meta-classes executing at import time
Lastly, i see metaprogramming at work not only when coding in a computer language. Discussing the legal framing for executing programs on the internet is some kind of metaprogramming, especially if you consider licensing and laws as human-interpreted code which affects how programs can be written, constructed and executed. In reverse, web applications increasingly affect how we interact with each other other, thereby implementing rules formerly dealt with in the arena of politics. Therefore, metaprogramming and politics are fundamentally connected topics.
have metafun, i. e. take fun stuff as input to generate more of it holger
While many agree that global state is evil, the so called “thread locals” are not much better. Even though they help to separate state on a per-thread or per-greenlet basis, they still are global within that context. In particular (thread) global state means that:
- Invoked functions can change bindings of an invoking function as a side effect
- thread locals may linger around even if their state is not used or became invalid
Meet “execution locals” which avoid these problems. Find the code released on PyPI:
It’s some 60 lines of code and tested on python2.5 up to python3.3 and pypy and ready to be played with. I inline its README.txt below in case you can’t or don’t want to switch reading context. One more note: If I were to design a new language i’d probably remove “globals” all together and only offer something like the “xlocal” type with a more straight forward syntax.
execution locals: killing global state (including thread locals)
The xlocal module provides execution locals aka “xlocal” objects which implement a more restricted variant of “thread locals”. An “xlocal” instance allows to manage its attributes on a per-execution basis in a manner similar to how real locals work:
- Invoked functions cannot change the binding for the invoking function
- existence of a binding is local to a code block (and everything it calls)
Attribute bindings for an xlocal object will not leak outside a context-managed code block and they will not leak to other threads or greenlets. By contrast, both process-globals and “thread locals” do not implement these properties.
Let’s look at a basic example:
# content of example.py from xlocal import xlocal xcurrent = xlocal() def output(): print "hello world", xcurrent.x if __name__ == "__main__": with xcurrent(x=1): output()
If we execute this module, the output() function will see a xcurrent.x==1 binding:
$ python example.py hello world 1
Here is what happens in detail: xcurrent(x=1) returns a context manager which sets/resets the x attribute on the xcurrent object. While remaining in the same thread/greenlet, all code triggered by the with-body (in this case just the output() function) can access xcurrent.x. Outside the with- body xcurrent.x would raise an AttributeError. It is also not allowed to directly set xcurrent attributes; you always have to explicitely mark their life-cycle with a with-statement. This means that invoked code:
- cannot rebind xlocal state of its invoking functions (no side effects, yay!)
- xlocal state does not leak outside the with-context (lifecylcle control)
Another module may now reuse the example code:
# content of example_call.py import example with example.xcurrent(x=3): example.output()
which when running …:
$ python example_call.py hello world 3
will cause the example.output() function to print the xcurrent.x binding as defined at the invoking with xcurrent(x=3) statement.
Other threads or greenlets will never see this xcurrent.x binding; they may even set and read their own distincit xcurrent.x object. This means that all threads/greenlets can concurrently call into a function which will always see the execution specific x attribute.
Usage in frameworks and libraries invoking “handlers”
When invoking plugin code or handler code to perform work, you may not want to pass around all state that might ever be needed. Instead of using a global or thread local you can safely pass around such state in execution locals. Here is a pseudo example:
xcurrent = xlocal() def with_xlocal(func, **kwargs): with xcurrent(**kwargs): func() def handle_request(request): func = gethandler(request) # some user code spawn(with_xlocal(func, request=request))
handle_request will run a user-provided handler function in a newly spawned execution unit (for example spawn might map to threading.Thread(…).start() or to gevent.spawn(…)). The generic with_xlocal helper wraps the execution of the handler function so that it will see a xcurrent.request binding. Multiple spawns may execute concurrently and xcurrent.request will carry the execution-specific request object in each of them.
Issues worth noting
If a method decides to memorize an attribute of an execution local, for example the above xcurrent.request, then it will keep a reference to the exact request object, not the per-execution one. If you want to keep a per-execution local, you can do it this way for example:
Class Renderer: @property def request(self): return xcurrent.request
this means that Renderer instances will have an execution-local self.request object even if the life-cycle of the instance crosses execution units.
Another issue is that if you spawn new execution units, they will not implicitely inherit execution locals. Instead you have to wrap your spawning function to explicitely set execution locals, similar to what we did in the above “invoking handlers” section.
Copyright / inspiration
This code is based on discussions with Armin Ronacher and others in response to a tweet of mine. It extracts and refines some ideas found in Armin’s “werzeug.local” module and friends.
MIT, see LICENSE for more details.
With this “xlocal” experiment i am trying to explore how much of new language ideas can be prototyped with existing Python. For more related goals, see my earlier post on If i were to design a new language.
I’d see to base syntax and semantics on Python3, but strip and rebase it:
- no C: implement the interpreter in RPython, get a JIT for free and implementation bits from PyPy’s Python interpreter (parsing, IO, etc.)
- no drags-you-down batteries: lean interpreter core and a standard battery distro which is tested against the last N interpreter versions + current
- no yield: use greenlets to implement all of what yield provides and more
- no underlying blocking on IO: base it all on event loop, yet provide synchronous programming model through greenlets
- no c-level API nor ctypes: use cffi to interface with c-libraries
- no global state: just support state bound to execution context/stack
- no GIL: support free threading and Automatic Mutual Exclusion for dealing with shared state
- no setup.py: have a thought-through story and tools from the start for packaging, installation, depending/interfacing between packages
- no import, no sys.modules: provide an object with which you can access other packages’s objects and introspect/interact with one’s own package
- no testing as an afterthought: everything needs to be easily testable, empowered assert statement and branch-coverage supported from the core.
- no extensibility as an afterthought: support plugins and loose coupling through builtin 1:N calling mechanism (event notification on steroids)
- no unsafe code: support IO/CPU/RAM sandboxing as a core feature
- no NIH syndrome: provide a bridge to a virtualenv’ed Python interpreter allowing to leverage existing good crap
Anything else? Probably! Discussion needed? Certainly. Unrealistic? Depends on who would participate — almost all of the above has projects, PEPs and code showcasing viability.
Btw, did you know that when we started PyPy we initially did this under the heading of “Minimal Python”? Some of the above ideas above and their underlying motivations were already mentioned when I invited to the first PyPy sprint almost 10 years ago:
I learned since then that Python has more complex innards than it seems but i still believe it could be both simpler and more powerful.