平行点安装 [英] Parallel Pip install

问题描述

我们的Django项目越来越庞大.我们有数百个应用程序，并使用大量的第三方python软件包，其中许多都需要编译C.当我们需要为主要版本创建新的虚拟环境时，我们的部署会花费很长时间.话虽如此，我希望从Pip开始加快工作速度.有谁知道会并行安装软件包的Pip分支?

Our Django project is getting huge. We have hundreds of apps and use a ton of 3rd party python packages, many of which need to have C compiled. Our deployments are taking a long time when we need to create a new virtual environment for major releases. With that said, I'm looking to speed things up, starting with Pip. Does anyone know of a fork of Pip that will install packages in parallel?

到目前为止我已经采取的步骤:

Steps I've taken so far:

• 我一直在寻找一个仅能成功完成此任务的项目.我确实找到了这个Github要点: https://gist.github.com/1971720 ，但结果与我们的结果几乎完全相同单线程朋友.

• I've looked for a project that does just this with little success. I did find this Github Gist: https://gist.github.com/1971720 but the results are almost exactly the same as our single threaded friend.

然后，我在Github上找到了Pip项目，并开始在分支网络中进行查找，以查看是否可以找到提及所做的任何提交.那里很乱.我会分叉它，如果需要的话，我自己尝试并行化它，我只是想避免花时间去做.

I then found the Pip project on Github and started looking through the network of forks to see if I could find any commits that mentioned doing what I'm trying to do. It's a mess in there. I will fork it and try to parallelize it myself if I have to, I just want to avoid spending time doing that.

我在ep.io的DjangoCon 2011上看到了一个演讲，解释了他们的部署内容，他们提到了并行化pip，传送.so文件而不是编译C和镜像Pypi，但是他们没有谈到它们是如何做到的.或他们使用的是什么.

I saw a talk at DjangoCon 2011 from ep.io explaining their deployment stuff and they mention parallelizing pip, shipping .so files instead of compiling C and mirroring Pypi, but they didn't touch on how they did it or what they used.

推荐答案

您是否分析了部署过程以了解实际时间?令我感到惊讶的是，运行多个并行的pip进程并不能加快速度.

Have you analyzed the deployment process to see where the time really goes? It surprises me that running multiple parallel pip processes does not speed it up much.

如果花时间查询PyPI和查找软件包(尤其是当您还从Github和其他来源下载时)，那么设置自己的PyPI可能会有所帮助.您可以自己托管PyPI，并将以下内容添加到您的requirements.txt文件中( docs ):

If the time goes to querying PyPI and finding the packages (in particular when you also download from Github and other sources) then it may be beneficial to set up your own PyPI. You can host PyPI yourself and add the following to your requirements.txt file (docs):

--extra-index-url YOUR_URL_HERE

或以下，如果您希望完全替换官方PyPI:

or the following if you wish to replace the official PyPI altogether:

--index-url YOUR_URL_HERE

这可能会加快下载速度，因为现在所有软件包都在附近的计算机上找到.

This may speed up download times as all packages are now found on a nearby machine.

很多时间还需要使用C代码来编译软件包，例如PIL.如果发现这是瓶颈，那么值得研究多个进程中的代码.您甚至可以在计算机之间共享编译的二进制文件(但是许多事情都需要类似，例如操作系统，CPU字长等等)

A lot of time also goes into compiling packages with C code, such as PIL. If this turns out to be the bottleneck then it's worth looking into compiling code in multiple processes. You may even be able to share compiled binaries between your machines (but many things would need to be similar, such as operating system, CPU word length, et cetera)

这篇关于平行点安装的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！