PEP438 is live: speed up python package installs now!
My “speed up pypi installs” PEP438 has been accepted and transition phase 1 is live: as a package maintainer you can speed up the installation for your packages for all your users now, with the click of a button: Login to https://pypi.python.org and then go to urls for each of your packages, and specify that all release files are hosted from pypi.python.org. Or add explicit download urls with an MD5. Tools such as pip or easy_install will thus avoid any slow crawling of third party sites.
Many thanks to Carl Meyer who helped me write the PEP, and Donald Stufft for implementing most of it, and Richard Jones who accepted it today! And thanks also to the distutils-sig discussion participants, in particular Phillip Eby and Marc-Andre Lemburg.
Written by holger krekel
May 19, 2013 at 7:49 am
Posted in metaprogramming
17 Responses
Subscribe to comments with RSS.
metaprogramming and politics 
Good to know!
Can you clarify a bit what option to use in PyPI? I looked through my packages, but couldn’t find which option. (I typically release packages using `./setup.py sdist register upload`)
Diederik van der Boor
May 19, 2013 at 9:08 am
You can just go to the “urls” tab and change to “do not extract urls”. That’s all, no need to deal with the “file urls” at all to achieve the speed up for installers. See http://merlinux.eu/~hpk/pypi1.png for an example of what is needed.
holger krekel
May 19, 2013 at 9:14 am
Here’s a guide on how to update your packages http://pypi-externals.caremad.io/help/what/ .
Donald Stufft (@dstufft)
May 19, 2013 at 3:14 pm
Just heard about this, that’s a really good initiative !
Up till now, providing a comprehensive doc with links in long description used to punish our users…
I did not find a “remove all” button, though, for the bunch of irrelevant links I have on my packages.
Thanks.
Georges Racinet
May 19, 2013 at 4:23 pm
Usually those URLs do no harm, though. pip/easy_install don’t consdier them unless they look like a package archive. The main thing that makes installs slow, are/were the links with “rel=’homepage'” or “rel=’download'” which caused pip/easy_install to crawl the link and look for more links there.
holger krekel
May 19, 2013 at 4:38 pm
Thanks for that precision
gracinet
May 20, 2013 at 2:31 pm
IMHO, crawling Web pages, or HTML index pages to find download links is a bad design. I can’t understand why al this can’t rely on pure REST or XMLRPC based protocols.
Gilles Lenfant Lenfant
May 20, 2013 at 2:25 pm
That and the freaking case independency / mispelle correction relying on pip reloading the whole ‘simple/’ page (actually the main pypi has a 301 redirect, but mirrors may not).
My guess is that it was a necessity for wide adoption of the PyPI a while ago. Not really sustainable in the long run, though.
gracinet
May 20, 2013 at 2:31 pm
Misspelled ‘misspell’ !
gracinet
May 20, 2013 at 2:33 pm
sure, if you read pep438 you find the historic reasons behind this. The PEP tries to address and move away from it in a backward-compatible way.
holger krekel
May 20, 2013 at 7:47 pm
Er, HTML + HTTP is a REST Api.
Donald Stufft (@dstufft)
May 22, 2013 at 3:04 pm
HTML has poor custom data semantic support, and mixing HTML with business oriented microformats overcharges the payload and makes it difficult and slow to build and parse.
I was talking of *real* REST leveraging GET/POST/PUT/DELETE HTTP verbs and dedicated JSON or XML payloads.
Gilles Lenfant
September 21, 2013 at 8:15 am
Holger, thanks you for writing PEP438! You just pushed one of my main objections against Python package installations (third party distribution sites) towards deprecation 🙂
peterodding
May 20, 2013 at 6:44 pm
Great, updated the packages I have access to right away !
Thanks Holger !
Harro van der Klauw (@hvdklauw)
May 22, 2013 at 5:25 am
Generally I don’t learn article on blogs, but I wish to say that this write-up very compelled me to take a look at and do it! Your writing style has been amazed me. Thank you, very great post.
text3
May 26, 2013 at 4:50 am
What’s Going down i am new to this, I stumbled upon this I’ve discovered It positively
useful and it has aided me out loads. I hope to give a contribution
& help different customers like its helped me. Great job.
watch college football online for free streaming
September 13, 2013 at 7:15 pm
When some one searches for his necessary thing, so he/she needs to be available that in detail, so that thing is maintained
over here.
email
September 21, 2013 at 7:31 am