Julia并行编程 - 使所有工作人员都可以使用现有功能 [英] Julia parallel programming - Making existing function available to all workers

问题描述

我遇到以下问题：

我有一个称为 TrainModel 的函数，它在单个线程上运行很长时间。当它完成计算后，它将返回一个函数作为输出参数，我们称之为 f 。当我查询这个 f 的类型时，Julia返回：
$ b

（我不清楚这最后一条信息是否对读者有用）

现在在第二步中，我需要将函数 f 应用于一个非常大的数组值。这是我想平行的一个步骤。已经用多个进程启动了Julia，例如

  julia -p 4 
  pre 
 
理想情况下，我会使用：
 $ b $ pre $ p $（p， my_values）
  

或许：

<$ p $对于ii = 1：100000000
f（my_values [ii]）
结束
aux = @parallel（hcat） p $ p>



不幸的是，这是行不通的。 Julia抱怨工作人员没有意识到功能 f ，也就是说我得到了一个信号：



错误：函数f在进程2中没有定义

如何使函数 f 可用于所有worker？很明显，一个脏的解决方案是在所有员工上运行耗时的功能 TrainModel ，比如：b


  @everywhere f = TrainModel（...）
  

但是这样做太浪费了当我想要的只是结果 f 是可用的所有工人。

虽然我搜索有类似问题的帖子，所以远远我找不到答案...

在此先感谢！
best，

N。

解决方案

返回函数看起来很优雅，但不幸的是，与JavaScript不同，Julia在创建函数时并没有解决所有的变量。
技术上讲，您的训练函数可以为所有训练过的参数生成具有文字值的函数源代码。然后将其传递给每个工作进程，这些工作进程可以在其环境中将其解析为一个可调用的函数。

我建议返回一个包含所有信息的数据结构产生训练的函数：ANN的权重，支持向量，决策规则...
在工作者进程上定义一个受过训练的函数，以便它将使用训练过的参数。您可能希望有能力将训练结果保存到磁盘，以便您轻松地重新生成计算结果。

I am faced with the following problem:

I have a function called TrainModel that runs for a very long time on a single thread. When it finishes computing, it returns a function as an output argument, let's call it f. When I enquire the type of this f, Julia returns:

(generic function with 1 method)

(I am not sure of this last piece of information is useful to anyone reading this)

Now in a second step, I need to apply function f on a very large array of values. This is a step that I would like to parallelise. Having had started Julia with multiple processes, e.g.

julia -p 4

ideally, I would use:

pmap(f, my_values)

or perhaps:

aux = @parallel (hcat) for ii=1:100000000
        f(my_values[ii])
      end

Unfortunately, this doesn't work. Julia complains that the workers are not aware of the function f, i.e. I get a messsage:

ERROR: function f not defined on process 2

How can I make function f available to all workers? Obviously a "dirty" solution would be to run the time-consuming function TrainModel on all workers, like this perhaps:

@everywhere f = TrainModel( ... )

but this would be a waste of cpu when all I want is that just the result f is available to all workers.

Though I searched for posts with similar problems, so far I could not find an answer...

Thanks in advance! best,

N.

解决方案

The approach to return the function seems elegant but unfortunately, unlike JavaScript, Julia does not resolve all the variables when creating the functions. Technically, your training function could produce the source code of the function with literal values for all the trained parameters. Then pass it to each of the worker processes, which can parse it in their environment to a callable function.

I suggest to return a data structure that contains all the information to produce the trained function: weights of an ANN, support vectors, decision rules ... Define a the "trained" function on the worker processes, such that it will utilized the trained parameters. You might want to have the ability of saving the results of the training to disk anyway, so that you can easily re-produce your computations.

这篇关于Julia并行编程 - 使所有工作人员都可以使用现有功能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！