conda是否安装了线程安全操作？ [英] Is conda install a thread-safe operation?

问题描述

我想将软件包安装到多个conda环境中。一个接一个地执行此操作需要花费一些时间，因此，如果我可以为每个环境并行运行所有 conda install 步骤，那将是很好的。尝试并行运行conda时，这是否可能或有冲突（可能与硬链接和锁定文件有关）？

I would like to install packages into multiple conda environments. Doing this one after the other takes quite some time, so it would be nice if I could run all the conda install steps for each environment in parallel. Would this be possible or are there conflicts (relating to hard links and lock files, possibly) when trying to run conda in parallel?

推荐答案

简短的答案：不，它不能同时运行。

大多数Conda处理交易安全的方式都建立在v4版本中。 3。 发行说明在v4.3.0版本中中，有关更改锁的内容明确地注释了运行多个进程的情况：

Most of how Conda handles transaction safety was established in version v4.3. The release notes in v4.3.0 regarding changes to locks explicitly comment on running multiple processes:

[U]警告用户，未定义的行为当conda在多个进程中运行并且在相同的程序包缓存和/或环境中运行时，可能会导致结果。

[U]sers are cautioned that undefined behavior can result when conda is running in multiple process and operating on the same package caches and/or environments.

听起来像是谈论不同的环境，所以这不应该成为问题。但是，您需要确保要安装的软件包已下载到软件包缓存中，否则并不安全。

It sounds like you're talking about different environments, so that shouldn't be an issue. However, you need to ensure that the package(s) to be installed is already downloaded into the package cache, otherwise it is not safe.

有一个-仅下载标志，该标志只会将包添加到包缓存中（即部分无法同时完成）。但是问题在于，仍然需要在每个环境中完成此操作，因为不同的环境可能具有不同的约束（例如，不同的Python版本），并且需要使用不同的程序包构建。

There is a --download-only flag, which will only add the package to the package cache (i.e., the part that cannot be done concurrently). But the issue is that this would still need to be done on a per-env basis, since different envs could have different constraints (e.g., different Python versions) that require different builds of the package.

我认为您可以在CLI上做的最好的事情是

I think the best you could do at the CLI is

1. 运行 conda install-在每个env上依次下载仅下载pkg ，然后


2. 为env并行运行 conda install pkg

1. Run conda install --download-only pkg sequentially on each env, then
2. Run conda install pkg in parallel for the envs.

但是，这在任何官方建议中都不是，而且Conda进行交易方式的变化可能导致这种情况不安全。我还要说，我非常怀疑这会为您节省很多时间；实际上，这可能需要更长的时间。这种方法将涉及到每个环境都要解决和准备两次事务，这通常是计算量最大的步骤。最终并行化的部分涉及磁盘事务，该事务将受I / O约束，所以我有点怀疑会节省任何时间。

This is, however, not in any official recommendation, and changes in how Conda does transactions could lead to this not being safe. I'll also say that I very much doubt this will save you much time; in fact, it might take longer. This approach will involve every env having to solve and prepare transactions twice, and that is usually the most computationally intensive step. The part you end up parallelizing involves disk transactions, which is going to be I/O bound, so I kind of doubt any time will be saved.

虽然这不能肯定地证明其安全性，但是我们可以显式检查事务以确保当我们执行上述步骤2时，它仅涉及LINK事务

While this doesn't positively prove its safety, we can explicitly examine the transactions to make sure that when we run Step 2 above, it will only involve LINK transactions.

为了测试这一点，我做了两个环境：

To test this, I made two envs:

conda create -n foo -y python=3.6
conda create -n bar -y python=3.6

然后我检查输出

conda install -n foo -d --json pandas

其中显示了FETCH和LINK事务的列表。前者涉及操纵程序包缓存，而后者仅涉及env。如果再运行

which shows a list of both FETCH and LINK transactions. The former involve manipulating the package cache, whereas the latter only the env. If I then run

conda install -n foo --download-only pandas

并再次检查，

conda install -n foo -d --json pandas

我现在仅看到LINK交易。值得注意的是， -n bar 现在也是如此，这应进一步强调步骤1应该按顺序进行的事实。好消息是，它不会导致重新下载相同的程序包；不利的一面是，它涉及到每个环境中都存在的解决方案。在更加异构的环境中，我们可以预期每个环境中可能会有不同的FETCH操作。

I now see only LINK transactions. Notably, the same is now true for -n bar, which should reinforce the fact that Step 1 should be done sequentially. The good part is that it won't lead to redownloading the same package; the bad part, that it involves a solve happening in every env. In a more heterogenous environment, we could expect there might be different FETCH operations in each env.

最后，我可以运行并行最终安装

Finally, I can run the parallel final install

conda install -n foo -y pandas & conda install -n bar -y pandas &

这是安全的，如果我们可以假定不同环境中的LINK交易是安全的。

which is safe if we can assume that that LINK transactions in different envs are safe.

这篇关于conda是否安装了线程安全操作？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！