metaprogramming and politics

Decentralize. Take the red pill.

Message to my IT/hacking friends (Mar17)

with one comment

Some thoughts and advises on March 17 from BB33, a little office and hackerspace in Freiburg in the black forest. Sitting here alone. My 8yo and partner are good, a few streets further, as is her family, for now. My sisters and many other friends are less well but i won’t detail this here. School has ended but my 8yo is totally angry with Covid-19 … is asking if it could be killed by throwing a host of atomic bombs on it … to which my answer is: “nuclear power can not kill covid-19 even it would kill all human life on the planet. But yes, sincerely, i understand your frustration — let’s take a bicicly ride together.”

If you are like me and many of my friends you’ll get a host of demands because suddenly remote learning and working is in so many minds, also minds who have access to money while others are in urgent need. How to react and how to care?

TLDR; care less about work and exams and office work. Covid-19 induces planetary-scale changes and we need to act situational, not blindly bound by rules but with a caring attitude and focused on our neighborhoods, city and regional-level happenings.

How are you today?

How is your family, friends and the many strangers of your physical neighborhood? Can you personally talk to younger and older people to help them navigate and adjust to the new paraonia-normal? This is a time of massive fears and we need to talk with each other, 1:1 and in small groups and with strangers. Find and give emotional backing.

Now i didn’t talk about you yet … because i don’t know you and i’ll probably never get to know you, certainly never fully. I don’t even know myself fully. Each one of us is unique and has a unique situation — some have a sister who just went through intensive care because of lung problems, another has caring duties for someone with dementia or works in a care-home. Yet others have lost relatives in a different country who are experiencing far worse troubles and fear than you are. Use the phone if you can, make a call for those you care for. Don’t be angry with people who are upset and in fear. Relate humanly.

Take Breaks from the digital, repetitively

Take breaks from the digital. There are shitloads of news and now is the time to stop this bad habit of “news binging”, of trying to not miss a thing. You will miss tons of things — the question is: are you aware of and empathic with situations around you, first neighbors around your own living place, then the quarter, the city, the region and other states? You will also get filtered news from friends — try to make sure it’s scientifically rooted info, filter out bullshit before your communicate it to others.

The biggest challenge for most of us is an emotional one and dealing with fears. This will come in waves and be unevenly distributed. You can not constantly care for others, though. Try to establish rhythms of online/offline-ness that are healthy for your emotional state. Help others to re-arrange. Nobody has lived through a crisis at this planetary scale.

Avoid “White knight” syndrome and profit motives

Try to get rid of the “White Knight” syndrome and the feeling of urgency to save the world. This is a deep running emotional arrangement especially with us male folks. And then we want to be applauded and seen as heroes or winners. This approach is dangerous on a planetary scale. Don’t think about platforms and how to promote your project now. On the other hand, onboard those who want to help with your resilience and decentralization and care projects.  Resist urgency, care for yourselves and others.

Question your motivation if you or colleagues now are “out to make a profit” while disregarding the catastrophe unfolding around them. Just because you are physically safe and well-prepped doesn’t mean you’ll have a nice live if people and caring collapses around you. Those “profiteering” people will not be looked on nicely for the times to come.

Stop Community transmissions, soften police shutdowns

It’s not easy for anybody to stop habits and doing what you did just a few days ago. Don’t wait for orders from above before you move. Don’t take it from me but take it from the WHO Executive director who has personally crisis-managed multiple Ebola-outbreaks:

https://twitter.com/ConallMcD/status/1238865202868883461

It doesn’t matter how dangerous Covid-19 really is — we can not wait to first be sure of what is going to happen before we move. You can reasonably assume Covid-19 is deadly for many and you are transmitting it, the young folks who want to party are transmitting it, and people in the bakery next door who cling to their practice of walking into a tight shop situation, are transmitting it. And the logistics people who are bringing your remotely ordered packages, are transmitting it. Stay distant, teach others.

Try to help yourselves and others. All collective grouping and action is now based on social distancing (IOW: on respecting people’s safe spaces and demanding respect). However, the many people who care for the ill, work in critical logistics, operate power plants, telecommunications and water physical infrastructure, fire departments etc. … they all can not avoid to get into contact with others. Some of them are fighting in a war-like situation. Support them however you can. Don’t hoard, but share and help.

You might do a walk with a dear friend — just keep distance. If others don’t manage to do it, including groups of young kids, or people in disarray, or people clinging to their work because of fears and habit … empathize with them, help them to re-arrange. Suggest other community action than drinking and sitting together closely. This is all not only about cleaning your hands but looking out for others (young and old around you, physically) who cling to social practices that help Covid-19 spread.

If we can’t stop community transmission the government will need to do a partial/total police enforced lockdown. This would be a pity because it’s great to be able to walk out alone or with a socially distant yet dear friend, even make some music together etc.

For others like my italian friends it’s already too late, they are locked into their flats. But they sing on the balconies together — there is always a human, caring and playful option left in any crisis situation!

Question your work place and what you can do now

Re-evaluate what remains important or what new importances emerges. A few random takes:

  • support your local infrastructure: health and caring folks, sellers of medicine and food, those who transport away your garbage, homeless people — don’t look at it from the angle of regulations but of actual situations. Can the hotels who lost all tourists be talked to to accommodate homeless people and help them arrange in a socially distant manner?
  • Can hotels and other free facilities be prepared to accommodate refugees? In Europe we need to find a way to help Greece and the refugee situation there — the Merkel Gov promised to fly 1500 unaccompanied children — let’s designate an area where they can be quarantined and cared for. This would be a signal also against the uncaring right wing who will sooner or later try to pin blame on “the people abroad”.  Sidenote: I feel sorry for those folks who currently have to deal with authoritarianism (right or left wing). My best remote advise is: try to ignore and work around, move underneath and in between. Look out for co-operation and not for fights.
  • Covid-19 is pretty egalitarian and doesn’t care very much if you are rich or poor, white, brown, yellow, green or black … male or female. Here it will be crucial to overcome what is called “social darwisnism” because nobody wants to live in a world driven by acceptance and ignorance of old, disadvantaged or fleeing people dying  miserably. Consider helping to free prisoners who are in only for minor things and maybe take them to hotel rooms with a phone. Imagine yourselves in a prison being shut out from relatives and friend visits and calls, in the current situation.
  • Can you talk with city and regional governments folks and see what you can arrange with them? If you find un-cooperative people ignore them, don’t fight them. Save your energy for useful co-operations. Political reckonings can wait.
  • Is your work that you have been doing last week still relevant? Will somebody existentially suffer if you don’t go to the office, don’t preapre for the next deadline? Many of us knew it all along: officing work is overrated, caring work is underrated. There is no need to get theoretical or intellectual anymore about this: just look around and think yourself.
  • “I told you so” attitudes are not helpful yet it’s so hard to get rid of it. Talk about the future and not the past because it is your own choices with the people around you that will shape the futures to come.
  • Don’t fight over principles or rules or schedules. Try to evolve within your own situations and in the situations you find yourself in. Do 1:1 or small-group calls and don’t wait much for orders from above or from the assembly. The “above” has a tendency to try to fixate the past, to secure the future, but the future is already here, yet unevenly distributed on a planetary scale.
  • Move and act and communicate before you can be sure. Don’t wait to be right or for decisions from above before you move. Relax and care for yourself. You are not the only one to be emotionally exhausted.

Some things end, many more are beginning

Take care, stay socially distant, and develop better caring practices.  “Socially distant” can means more proximity as we start to actually look around more carefully. We can get through this as better and more healthy people, on a more livable planet, with more sensible and caring attitudes — if we and the still functioning parts of governments can avert even larger catastrophes than Covid-19 — pending nuclear reactor meltdowns, financial meltdowns, tele-communications outages, larger scale wars. Disaster planning, worst-case preparedness,  caring and moving early … this is the new game. Never forget to rest.

Playing a “profiteering” game with the lifes of people is, most of realize in their faces now, a questionable stance. Cars, planes and global busynesses just hit the wall and we don’t know yet know how we re-arrange the social, the warez, the traffic, the financials, the electrical and other power systems and the caring and critical infrastructure work. A return to the previous known normal will not happen. Accept and embrace.

But let’s please all see that no nuclear reactor goes bust now because we would all otherwise see the current times as “peanuts” compared to what would happen then. Do your nearby nuclear reactor operators have sufficient local agency to shut down their plants or do they need to wait for some OK from a central place that might never come in time? Do you want to find out? Nobody is in this as a spectator who sits back and judges and then votes in an election.  Consumerism must end and evolve to something different. Everybody bodily feels they are a participant now in something that nobody can yet determine what it will become. Trust that there are many others who think and care the same …

The animals and plants and forests are certainly feeling a big change as humans, finally, start to change their habits … The two birds who are nesting high up in a nearby Seqouia in the street where i live, the little animals who suddenly walk around freely at night because the streets are empty … i imagine the ecologies around us to experience the situations as a big relieve and many species are curious what is happening next. So am i and trust that our extinction rebellions are co-operative in nature and become cross-special. Stay safe and breathe. And by all means lough! Joy and joking and playing can help get us through. And Rest. Take care.

Written by holger krekel

March 17, 2020 at 9:37 am

Posted in metaprogramming

Going beyond codes of conduct

with one comment

When someone utters something that turns out to not be acceptable to many, what do you do? One way out is to punish, to try exclude that someone according to some pre-established rules. It’s an action that should be used sparingly and with precise reasoning as it otherwise has chilling effects and causes alienation.  You might get people to avoid certain behaviours in public but sentiments are still going to flow in private discussions.
The other possibility is to treat “incidents” as an opportunity to engage with underlying reasons and questions. Here are some of mine when it comes to the believe that one gender is superior to another, in the context of the programming communities i am connected with:

  • To begin with, why are many male programmers working 40 hours a week while their wife cares at home for children? See Traditional family models in the IT and Python world for more on this question.
  • How does it come to biases: When i imagine a “genius”, who do i think of? When i hear “keynote speaker” whom do i envision? At PyUnconf 2014 in Hamburg i discussed relations between centralized technical systems and normative thinking. Often, these norms are felt as “natural” or “true”.  But we are not only producing thoughts, our thoughts are products of cultural mass processes and less individual than they feel.
  • Who is speaking in the name of a community? Who is the “we”? And if “everyone” can participate, who actually participates? Who gets to decide, who defines what is being talked about and how? Which norms and values play a role in the communication around all these issues?
  • Why are most programming communities composed of a majority of white males? Biology? Or is it the result of centuries old patriarchies and colonialisms in action? When you try to “undo” this, do you just impose some rules by which everyone should behave? Or do you actively work to raise awareness, discuss some of the fundamentals and inherited impregnations?

Written by holger krekel

April 9, 2015 at 10:42 am

pytest development reorganization, adopt pytest month!

leave a comment »

pytest1Today I went live with shifting pytest development to be more community driven. During a discussion at FOSDEM 2015 a significant subset of pytest contributors decided to create two organisations at github and bitbucket and shift core pytest and several plugins to them, on the initial suggestion of Anatoly Bubenkoff.  See the new pytest contribution page for details.  The teams currently have a dozen members and we are looking forward to integrate more contributors which all get full access and commit rights to all repositories, can push to the website and release to pypi.

Also at FOSDEM 2015, core pytest contributor Brianna Laugher suggested and started the Adopt pytest month initiative which will bring together pytest contributors, users and Open Source projects interested to use pytest in their project.  Many pytest contributors and practioners will participate which means you get excellent support for bringing your testing efforts up to speed with pytest.

The pytest team is also working towards a pytest-2.7 release, take a peak at the current changelog.  If you like to get something in, now is a good time to submit a pull request.  Or, if you are working with a company you may contract merlinux which in turns contracts contributors to quickly resolve any issues you might have or organises in-house training or consulting.  Apart from the direct benefit for your company it’s also a good way to support a sustained and ever-improving testing infrastructure commons for Python (pytest, tox, devpi projects in particular).

Some pytest core contributors

pytest core contributors at FOSDEM 2015 in Bruxelles, left to right:  Andreas Pelme, Floris Bruynooghe, Ronny Pfannschmidt, Brianna Laugher, Holger Krekel, Anatoly Bubenkoff

Written by holger krekel

February 27, 2015 at 2:56 pm

Posted in metaprogramming

Tagged with , ,

Thoughts on arguing end-to-end crypto and surveillance

with 24 comments

Many Western governors are pushing for laws mandating all private communication can be secretly read and analyzed for them.The latest attack targets the one technology that still enables some privacy on a massively surveilled internet: end-to-end encryption. As hackers or IT people we can not afford to lament that the public doesn’t understand the significance of end-to-end crypto or privacy if we don’t appreciate its value for societies at home and abroad ourselves.

Responding to the renewed surveillance attacks with quick technical or narrow economic counter arguments is not going to work. An appropriate response needs to consider the political history and context of the current crypto and surveillance debates. Moreover, to stem the never-ending waves of new secret agency laws a re-framing of the common security debates is crucial to avoid the never-ending succession of new powers for government.

Let me start by rejecting the idea that governmental surveillance attacks have anything to do with fighting ruthless killers (“terrorists”) however often this claim is repeated in broadcast media. This is not to disregard the power of repetition, see the endlessly repeated claims of the existence of “Weapons of mass destruction” as a pretext for the Iraq war, or the fact that advertisements work. But despite endless repetition, governmental surveillance attacks don’t have anything to do with fighting terrorists. To turn it around, and i think the burden ought to be on the framers, where is the hard evidence that mass surveillance of civilians has significant effect, if any, on preventing terrorist attacks against civilians? And even if surveillance would prevent a few attacks how would it compare to the dangers of more government power?

The “fight terrorists with surveillance” discussion framing is seriously flawed also for another reason. Within it you are always going to lose the argument against more surveillance. If not now then after the next terror event. Because proponents can always argue they were right: if no attack happens it proves surveillance works and we need more of it. If an attack happens it also proves we need more surveillance. In this framed logic there can never be any rolling back of government powers.

The way out is to unframe the discussion and discuss the political and historical contexts of “terror attacks” and “expanding surveillance” separately. Let’s start with surveillance. If fighting terrorism is a red herring what are the motivations and politics of expanding government surveillance?

Governors worry about their power base

Governors of all kinds worry that people decide to change things in ways which endanger the power their associated networks hold. And they are particularly afraid today because they know there are many reasons why people want to change things in more fundamental ways. As much as people have lost trust in governors, governors have lost trust into people to keep them and their ilk in power.

The fear of governors seems justified if you look at the example of Spain in 2015: big parts of Spain’s social movements associate with a very new party on the block: Podemos. It aims to win the election in December and currently is leading the polls against the two parties which have governed Spain since 1975. It could actually happen despite the German chancellor Merkel supporting the Spanish president Rajoy who just introduced draconian laws against protesters and is generally sending his troops everywhere to avert the decline of his power network. Having to resort to direct repression is a sign of lost political power and in the case of Spain, panic. If you remember that Spain is a major EU country it’s understandable that many other governors in the West are worried something similar might happen to them soon.

Governors are always afraid they could lose their sight and grip over what people in their constituency are up to. Today it is not enough to have buddies in broadcast media which frame the discussion and interpretation of events to the governor’s liking. You also need to understand and contain, if possible, wider internet discussions before they can effect change you don’t want. Governors learned from Hannah Arendt that private discussions form the basis for public opinions which in turn structure and determine governmental power. If that weren’t the case how could feminist and really any social struggle have succeeded? It certainly never was the broadcast media or governors who first talked about and demanded rights for women or other oppressed groups.

How to contain decentralized communication?

New realities are co-created in a more decentralized manner and quicker than ever. Communication platforms grew in the last decade because of the interests of people to communicate and connect with one another. Maybe that’s due to a lost sense of community in disintegrating city neighborhoods which make people use “social media”. But in any case, Youtube, Twitter, Gmail, Facebook and IOS/Android app platforms became big because they facilitated decentralized communication and sharing between people. This presents a problem to governors because web communications are harder to contain in acceptable ways.

For a typical broadcast media discussion format you can send allied experts and construct “position” and “opposition” and thus frame the discussion. For example, it’s acceptable to discuss the virtues and dangers of “press freedom”, how to deal with “islamist militants” or how to “defend our values and rights”. Western Governors find it much less acceptable to link the Hebdo killing of or the rise of the “Islamic State” to the recent Western wars in Iraq, Libya and Syria, or to the everyday killing of civilians through Western drones and torture. Governors can’t yet directly contain such unacceptable linking activities and they are worried about it. For the time being, they try to frame it as irrelevant and repeat the “we are being attacked by ruthless killers” on broadcast media some more. It still kind of works but it’s unclear for how long.

What helps to contain discussions is to implant “You are being watched!” into the minds of people discussing the future of their governance. Putting up some public examples of punishment for unacceptable dissent refines the message into “Watch your words (and internet links!)” … also known as internalized or self-censorship. That’s not just effective for governors in Saudi Arabia but for their Western allies as well. The recent US sentencing of journalist Barret Brown to 48 months of prison for posting a link to some leaked data on an IRC channel can be seen as an example of a public punishment with chilling effects.

Arguments and national tactics against crypto attacks

Governors have long realized they can exploit central communication platform ownership to tap into most private communications. But to their apparent shock, many IT companies in the Post-Snowden era are implementing decentralized encryption because they in turn want to assure users that they can’t surveil their private messages. As a reaction, governors are conspiring to prevent decentralized encryption reaching the masses which would see them losing their current in-depth access to private communication.  Psychologically speaking, losing power is always harder to accept than not having it in the first place.

A response to the crypto attacks which I consider optimistic, if not shallow, is “it’s not technically feasible to regulate or ban end-to-end crypto”. It underestimates the ability of governors to write laws which will drastically change the playing field even if in an incremental manner. To begin with, why shouldn’t it be possible to prevent companies from distributing apps which incorporate decentralized encryption? Google and Apple already employ their own regulation on what kind of apps are distributed through their stores. Another regulation on decentralized-crypto apps can probably be added by the governors in the US. And that would prevent decentralized encryption reaching the masses at least in the short term.

As to government access to end-to-end encryption, it’s true that backdooring crypto would make people more vulnerable against all kind of exploiting attacks, not just governmental ones. Governors might frame this dillema by claiming that security against physical attacks is more important than security against someone reading your messages. Such an argument already incorporates the flawed “it’s all about anti-terror” framing.  The increased vulnerability of everyone’s devices is a bit of a tricky issue for governors given they couldn’t protect their own data against Snowden. If neccessary, governors will try to make concessions. Some applications such as online banking could be allowed to use non-backdoored crypto. They have all the banking data already, anyway. They probably will want to exempt governmental communication itself as well. With that we’d end up with a complete reversal of the democratic principle: public governments to act in secret and private communication to be constantly surveilled.

Western Governors have learned from the last Cryptowars battles. They know full well that they can only break private communication encryption if they outlaw it in a synchronized international manner. Otherwise they would have a harder time to overcome national arguments like “companies are going to leave the country if you ban decentral encryption”. Therefore, we need to fend off attacks on decentralized crypto in at least some Western countries to make such commercial arguments useful. Concretely, US companies like Google and Apple will more strongly resist if the EU does not also illegalize decentralized crypto.

It is as crucial to prevent EU crypto regulations as it was two decades ago. During the crypto battles in the 1990ties I studied with the deeply inspiring Prof. Andreas Pfitzmann who consulted the German government on crypto regulation. Along with other colleagues and groups he tirelessly worked and finally turned the tides and prevented Germany and thus the EU from introducing government backdoors to crypto algorithms. This in turn lead France and then the US to drop their plans and eventually relax crypto export regulations to keep their companies competitive. Today, we are back to square zero and must again convince some EU governments or parliaments to refrain from crypto banning laws. It’s a fight we better not lose.

Lastly, I’d like to be clear if maybe controversial on the dreadful Anti-Terror topic: If the Western governments want to stop killers from targetting western individuals they first need to stop ruthlessly killing and terrorizing individuals from abroad. Nothing else will bring more physical security against terrorist attacks. It reminds me of the 2500 year old question from the chinese politician and philosopher Confucius: “The way out is via the door. Why is it that no one will use this method?”

Written by holger krekel

January 24, 2015 at 9:02 pm

Running tests against multiple devices/resources (in parallel)

with 2 comments

devicesHow to best distribute tests against multiple devices or resources with pytest? This interesting question came up during my training in Lviv (Ukraine) at an embedded systems company. Distributing tests to processes can serve two purposes:

  • running the full test suite against each device to verify they all work according to the test specification
  • distributing the test load to several devices of the same type in order to minimize overall test execution time.

The solution to both problems is easy if you use two pytest facilities:

  • the general fixture mechanism: we write a fixture function which provides a device object which is pre-configured for use in tests.
  • the pytest-xdist plugin: we use it to run subprocesses and communicate configuration data for the device fixture from the master process to the subprocesses.

To begin with, let’s configure three devices that are each reachable by a separate IP address. We create a list of ip addresses in a file:

# content of devices.json
["192.168.0.1", "192.168.0.2", "192.168.0.3"]

We now create a local pytest plugin which reads the configuration data, implements a per-process device fixture and the master-to-slave communication to configure each subprocess according to our device list:

# content of conftest.py

import pytest

def read_device_list():
    import json
    with open("devices.json") as f:
        return json.load(f)

def pytest_configure(config):
     # read device list if we are on the master
     if not hasattr(config, "slaveinput"):
        config.iplist = read_device_list()

def pytest_configure_node(node):
    # the master for each node fills slaveinput dictionary
    # which pytest-xdist will transfer to the subprocess
    node.slaveinput["ipadr"] = node.config.iplist.pop()

@pytest.fixture(scope="session")
def device(request):
    slaveinput = getattr(request.config, "slaveinput", None)
    if slaveinput is None: # single-process execution
        ipadr = read_device_list()[0]
    else: # running in a subprocess here
        ipadr = slaveinput["ipadr"]
    return Device(ipadr)

class Device:
    def __init__(self, ipadr):
        self.ipadr = ipadr

    def __repr__(self):
        return "<Device ip=%s>" % (self.ipadr)

We can now write tests that simply make use of the device fixture by using its name as an argument to a test function:

# content of test_device.py
import time

def test_device1(device):
    time.sleep(2)  # simulate long test time
    assert 0, device

def test_device2(device):
    time.sleep(2)  # simulate long test time
    assert 0, device

def test_device3(device):
    time.sleep(2)  # simulate long test time
    assert 0, device

Let’s first run the tests in a single-process, only using a single device (also using some reporting option to shorten output):

$ py.test test_device.py -q --tb=line
FFF
================================= FAILURES =================================
/tmp/doc-exec-9/test_device.py:5: AssertionError: <Device ip=192.168.0.1>
/tmp/doc-exec-9/test_device.py:9: AssertionError: <Device ip=192.168.0.1>
/tmp/doc-exec-9/test_device.py:13: AssertionError: <Device ip=192.168.0.1>
3 failed in 6.02 seconds

As to be expected, we get six seconds execution time (3 tests times 2 seconds each).

Now let’s run the same tests in three subprocesses, each using a different device:

$ py.test --tx 3*popen --dist=each test_device.py -q --tb=line
gw0 I / gw1 I / gw2 I
gw0 [3] / gw1 [3] / gw2 [3]

scheduling tests via EachScheduling
FFFFFFFFF
================================= FAILURES =================================
E   AssertionError: <Device ip=192.168.0.1>
E   AssertionError: <Device ip=192.168.0.3>
E   AssertionError: <Device ip=192.168.0.2>
E   AssertionError: <Device ip=192.168.0.1>
E   AssertionError: <Device ip=192.168.0.3>
E   AssertionError: <Device ip=192.168.0.2>
E   AssertionError: <Device ip=192.168.0.3>
E   AssertionError: <Device ip=192.168.0.1>
E   AssertionError: <Device ip=192.168.0.2>
9 failed in 6.52 seconds

We just created three subprocesses each running three tests. Instead of 18 seconds execution time (9 tests times 2 seconds per test) we roughly got 6 seconds, a 3-times speedup. Each subprocess ran in parallel three tests against “its” device.

Let’s also run with load-balancing, i.e. distributing the tests against three different devices so that each device executes one test:

$ py.test --tx 3*popen --dist=load test_device.py -q --tb=line
gw0 I / gw1 I / gw2 I
gw0 [3] / gw1 [3] / gw2 [3]

scheduling tests via LoadScheduling
FFF
================================= FAILURES =================================
E   AssertionError: <Device ip=192.168.0.3>
E   AssertionError: <Device ip=192.168.0.2>
E   AssertionError: <Device ip=192.168.0.1>
3 failed in 2.50 seconds

Here each test runs in a separate process against its device, overall more than halfing the test time compared to what it would take in a single-process (3*2=6 seconds). If we had many more tests than subproceses than load-scheduling would distribute tests in real-time to the process which has finished executing other tests.

Note that the tests themselves do not need to be aware of the distribution mode. All configuration and setup is contained in the conftest.py file.

To summarize the behaviour of the hooks and fixtures in conftest.py:

  • pytest_configure(config) is called both on the master and each subprocess. We can distinguish where we are by checking for presence of config.slaveinput.
  • pytest_configure_node(node) is called for each subprocess. We can fill the slaveinput dictionary which the subprocess slave can then read via its config.slaveinput dictionary.
  • the device fixture only is called when a test needs it. In distributed mode, tests are only collected and executed in a subprocess. In non-distributed mode, tests are run single-process. The Device class is just a stub — it will need to grow methods for actual device communication. The tests can then simply use those device methods.

I’d like to thank Anton and the participants of my three day testing training in Lviv (Ukraine) for bringing up this and many other interesting questions.

 

I am giving another such professional testing course 25-27th November at the Python Academy in Leipzig. There are still two seats available. Me and other trainers can also be booked for on-site/in-house trainings worldwide.

Written by holger krekel

November 12, 2013 at 7:43 am

Defeating Sauron with the “Trust on first use” principle

with 6 comments

photo from Alexandre Duret-Lutz Gandalf and Frodo did the right thing when they went for destroying the power of the all-seeing eye. The idea of a central power that knows everything undermines our ability to self-govern and influence important changes in society, it undermines a foundation of democracy.

As against Sauron, it seems like an impossible fight to try to protect our communication against present-day espionage cartels.  I see glimmers of hope, though. Certainly not much in the political space. Somehow our politicians are themselves too interested to use the eye on select targets — even if only the ones which Sauron allows them to see.

My bigger hope lies with technologists who are working on designing better communication systems. We still have time during which we can reduce Sauron’s sight. But to begin with, how do we prevent passive spying attacks against our communications?

A good part of the answer lies in the Trust on first use principle. The mobile Threema application is a good example: when two people first connect with each other, they exchange communication keys and afterwards use it to perform end-to-end encrypted communications. The key exchange can happen in full sight of the eye, yet the subsequent communication will be illegible. No question, the eye can notice that the two are communicating with unknown content but if too many of them do that this fact becomes less significant.

Of course, the all-seeying eye can send a Nazgul to stand in the middle of the communication to deceive both ends and listen in. But it needs to do so from the beginning and continously if it wants to avoid the victims from noticing. And those two can at any time meet to verify their encryption keys and would realize there  was a Nazgul-in-the-middle attack.

By contrast, both SSL and GPG operate with a trust model where we can hear Sauron’s distant laughter. The one is tied to a thousand or so “root authorities”, which can be easily reined in as need be. The other mandates and propagates such a high level of initial mistrust between us that we find it simply too inconvenient to use.

Societies and our social interactions are fundamentally build on trust. Let’s design systems which build on initial trust and which help to identify after-the-fact when it was compromised. If the eye has bad dreams, then i am sure massively deployed trust-on-first-use communication systems are among them.

Written by holger krekel

October 26, 2013 at 7:04 am

pytest-2.4.0: new fixture features, many bug fixes

with one comment

The just released pytest-2.4.0 brings many improvements and numerous bug fixes while remaining plugin- and test-suite compatible (apart from a few supposedly very minor incompatibilities). See below for a full list of details. New feature highlights:
  • new yield-style fixtures pytest.yield_fixture, allowing to use existing with-style context managers in fixture functions.
  • improved pdb support: import pdb ; pdb.set_trace() now works without requiring prior disabling of stdout/stderr capturing. Also the --pdb options works now on collection and internal errors and we introduced a new experimental hook for IDEs/plugins to intercept debugging: pytest_exception_interact(node, call, report).
  • shorter monkeypatch variant to allow specifying an import path as a target, for example: monkeypatch.setattr("requests.get", myfunc)
  • better unittest/nose compatibility: all teardown methods are now only called if the corresponding setup method succeeded.
  • integrate tab-completion on command line options if you have argcomplete configured.
  • allow boolean expression directly with skipif/xfail if a “reason” is also specified.
  • a new hook pytest_load_initial_conftests allows plugins like pytest-django to influence the environment before conftest files import django.
  • reporting: color the last line red or green depending if failures/errors occured or everything passed.

The documentation has been updated to accomodate the changes, see http://pytest.org

To install or upgrade pytest:

pip install -U pytest # or
easy_install -U pytest

Many thanks to all who helped, including Floris Bruynooghe, Brianna Laugher, Andreas Pelme, Anthon van der Neut, Anatoly Bubenkoff, Vladimir Keleshev, Mathieu Agopian, Ronny Pfannschmidt, Christian Theunert and many others.

may nice fixtures and passing tests be with you,

holger krekel

Changes between 2.3.5 and 2.4

known incompatibilities:

  • if calling –genscript from python2.7 or above, you only get a standalone script which works on python2.7 or above. Use Python2.6 to also get a python2.5 compatible version.
  • all xunit-style teardown methods (nose-style, pytest-style, unittest-style) will not be called if the corresponding setup method failed, see issue322 below.
  • the pytest_plugin_unregister hook wasn’t ever properly called and there is no known implementation of the hook – so it got removed.
  • pytest.fixture-decorated functions cannot be generators (i.e. use yield) anymore. This change might be reversed in 2.4.1 if it causes unforeseen real-life issues. However, you can always write and return an inner function/generator and change the fixture consumer to iterate over the returned generator. This change was done in lieu of the new pytest.yield_fixture decorator, see below.

new features:

  • experimentally introduce a new pytest.yield_fixture decorator which accepts exactly the same parameters as pytest.fixture but mandates a yield statement instead of a return statement from fixture functions. This allows direct integration with “with-style” context managers in fixture functions and generally avoids registering of finalization callbacks in favour of treating the “after-yield” as teardown code. Thanks Andreas Pelme, Vladimir Keleshev, Floris Bruynooghe, Ronny Pfannschmidt and many others for discussions.

  • allow boolean expression directly with skipif/xfail if a “reason” is also specified. Rework skipping documentation to recommend “condition as booleans” because it prevents surprises when importing markers between modules. Specifying conditions as strings will remain fully supported.

  • reporting: color the last line red or green depending if failures/errors occured or everything passed. thanks Christian Theunert.

  • make “import pdb ; pdb.set_trace()” work natively wrt capturing (no “-s” needed anymore), making pytest.set_trace() a mere shortcut.

  • fix issue181: –pdb now also works on collect errors (and on internal errors) . This was implemented by a slight internal refactoring and the introduction of a new hook pytest_exception_interact hook (see next item).

  • fix issue341: introduce new experimental hook for IDEs/terminals to intercept debugging: pytest_exception_interact(node, call, report).

  • new monkeypatch.setattr() variant to provide a shorter invocation for patching out classes/functions from modules:

    monkeypatch.setattr(“requests.get”, myfunc)

    will replace the “get” function of the “requests” module with myfunc.

  • fix issue322: tearDownClass is not run if setUpClass failed. Thanks Mathieu Agopian for the initial fix. Also make all of pytest/nose finalizer mimick the same generic behaviour: if a setupX exists and fails, don’t run teardownX. This internally introduces a new method “node.addfinalizer()” helper which can only be called during the setup phase of a node.

  • simplify pytest.mark.parametrize() signature: allow to pass a CSV-separated string to specify argnames. For example: pytest.mark.parametrize("input,expected",  [(1,2), (2,3)]) works as well as the previous: pytest.mark.parametrize(("input", "expected"), ...).

  • add support for setUpModule/tearDownModule detection, thanks Brian Okken.

  • integrate tab-completion on options through use of “argcomplete”. Thanks Anthon van der Neut for the PR.

  • change option names to be hyphen-separated long options but keep the old spelling backward compatible. py.test -h will only show the hyphenated version, for example “–collect-only” but “–collectonly” will remain valid as well (for backward-compat reasons). Many thanks to Anthon van der Neut for the implementation and to Hynek Schlawack for pushing us.

  • fix issue 308 – allow to mark/xfail/skip individual parameter sets when parametrizing. Thanks Brianna Laugher.

  • call new experimental pytest_load_initial_conftests hook to allow 3rd party plugins to do something before a conftest is loaded.

Bug fixes:

  • fix issue358 – capturing options are now parsed more properly by using a new parser.parse_known_args method.
  • pytest now uses argparse instead of optparse (thanks Anthon) which means that “argparse” is added as a dependency if installing into python2.6 environments or below.
  • fix issue333: fix a case of bad unittest/pytest hook interaction.
  • PR27: correctly handle nose.SkipTest during collection. Thanks Antonio Cuni, Ronny Pfannschmidt.
  • fix issue355: junitxml puts name=”pytest” attribute to testsuite tag.
  • fix issue336: autouse fixture in plugins should work again.
  • fix issue279: improve object comparisons on assertion failure for standard datatypes and recognise collections.abc. Thanks to Brianna Laugher and Mathieu Agopian.
  • fix issue317: assertion rewriter support for the is_package method
  • fix issue335: document py.code.ExceptionInfo() object returned from pytest.raises(), thanks Mathieu Agopian.
  • remove implicit distribute_setup support from setup.py.
  • fix issue305: ignore any problems when writing pyc files.
  • SO-17664702: call fixture finalizers even if the fixture function partially failed (finalizers would not always be called before)
  • fix issue320 – fix class scope for fixtures when mixed with module-level functions. Thanks Anatloy Bubenkoff.
  • you can specify “-q” or “-qq” to get different levels of “quieter” reporting (thanks Katarzyna Jachim)
  • fix issue300 – Fix order of conftest loading when starting py.test in a subdirectory.
  • fix issue323 – sorting of many module-scoped arg parametrizations
  • make sessionfinish hooks execute with the same cwd-context as at session start (helps fix plugin behaviour which write output files with relative path such as pytest-cov)
  • fix issue316 – properly reference collection hooks in docs
  • fix issue 306 – cleanup of -k/-m options to only match markers/test names/keywords respectively. Thanks Wouter van Ackooy.
  • improved doctest counting for doctests in python modules — files without any doctest items will not show up anymore and doctest examples are counted as separate test items. thanks Danilo Bellini.
  • fix issue245 by depending on the released py-1.4.14 which fixes py.io.dupfile to work with files with no mode. Thanks Jason R. Coombs.
  • fix junitxml generation when test output contains control characters, addressing issue267, thanks Jaap Broekhuizen
  • fix issue338: honor –tb style for setup/teardown errors as well. Thanks Maho.
  • fix issue307 – use yaml.safe_load in example, thanks Mark Eichin.
  • better parametrize error messages, thanks Brianna Laugher
  • pytest_terminal_summary(terminalreporter) hooks can now use “.section(title)” and “.line(msg)” methods to print extra information at the end of a test run.

Written by holger krekel

October 1, 2013 at 9:40 am

Posted in metaprogramming

Tagged with ,