在IJulia笔记本中的并行处理器上调用numpy [英] Calling numpy on parallel processors in IJulia notebook

问题描述

我想在IJulia笔记本中运行一个简单的代码，该笔记本使用python库numpy.我用PyCall调用numpy:

I want to run a simple code in the IJulia notebook which uses the python library numpy. I call numpy with PyCall:

使用PyCall

using PyCall

@pyimport numpy as np

@pyimport numpy as np

这很好.然后，我想将其拆分为几个处理器.我添加了处理器:

This works fine. I then want to split this up to several processors. I add the processors:

addprocs(4)

addprocs(4)

然后，我为函数f运行N/proc个迭代，其中proc是我的处理器数量.我将负载平均分配给计算机上的四个处理器:

Then, I run N/proc iterations for a function f, where proc is my number of processors. I split the load evenly between the four processors on my computer:

n=round(Int,N/proc);

proc_sum = @parallel (+) for i=1:proc

        f(n)

end

return proc_sum / proc

没有numpy，这可以正常工作.但是，当我尝试将numpy的代码拆分到不同的处理器时，会出现错误

Without numpy, this works fine. However, when I try to split the code with numpy to different processors, I get the error

错误(未处理的任务失败):在工作进程3上:

ERROR (unhandled task failure): On worker 3:

UndefVarError:未定义np

UndefVarError: np not defined

是否有其他方法可以使numpy在其他处理器上工作?请注意，我的茱莉亚(Julia)是0.5.2，我的树冠是.我知道以前曾有关于PyCall和Canopy的问题的报告，但我非常希望将Canopy保留在我的计算机上.

Is there any way to have numpy work on the other processors? Note that I have Julia 0.5.2, and I have Canopy. I know there have been issues reported before with PyCall and Canopy, but I would greatly prefer keeping Canopy on my machine.

推荐答案

要进一步扩展已经说过的内容，应在所有进程中加载​​所需的所有内容.例如:

To further expand on what has been said already, everything you need should be loaded in all the processes. E. g. :

addprocs(4) @everywhere using PyCall @everywhere @pyimport numpy as np

addprocs(4) @everywhere using PyCall @everywhere @pyimport numpy as np

您写的内容出错，因为所有进程都尝试使用@pyimport，但是只有主进程装载了PyCall.如果您需要许多软件包来进行计算，也许更简单的方法是在一个脚本(即load_modules.jl)中进行所有加载，然后简单地运行

What you wrote errored because all processes tried to use @pyimport but only the main process had PyCall loaded. If you require many packages to do your computations, maybe the easier is to do all the loading in one script, i.e. load_modules.jl and then simply run

addprocs(4) @everywhere include("load_modules.jl")

addprocs(4) @everywhere include("load_modules.jl")

using对于@everywhere似乎不是很健壮(已在Julia 0.6上修复，请参见

It seems that using is not very robust with @everywhere (fixed on Julia 0.6, see here). What seems to work better is:

addprocs(4) import PyCall @everywhere using PyCall @everywhere @pyimport numpy as np

addprocs(4) import PyCall @everywhere using PyCall @everywhere @pyimport numpy as np

这篇关于在IJulia笔记本中的并行处理器上调用numpy的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！