如何以及何时在 Julia 中使用 @async 和 @sync [英] How and When to Use @async and @sync in Julia

问题描述

我已阅读 文档>@async 和 @sync 宏，但仍然不知道如何以及何时使用它们，我也无法在互联网上的其他地方找到许多资源或示例.

I have read the documentation for the @async and @sync macros but still cannot figure out how and when to use them, nor can I find many resources or examples for them elsewhere on the internet.

我的近期目标是找到一种方法，让多个 worker 并行工作，然后等到他们都完成后继续执行我的代码.这篇文章:等待对于要在 Julia 中的远程处理器上完成的任务，包含一种成功的方法来完成此任务.我曾认为应该可以使用 @async 和 @sync 宏，但是我最初未能实现这一目标让我怀疑我是否正确理解如何以及何时使用这些宏.

My immediate goal is to find a way to set several workers to do work in parallel and then wait until they have all finished to proceed in my code. This post: Waiting for a task to be completed on remote processor in Julia contains one successful way to accomplish this. I had thought it should be possible using the @async and @sync macros, but my initial failures to accomplish this made me wonder if I am understanding properly how and when to use these macros.

推荐答案

根据 ?@async 下的文档，@async 将表达式包装在任务中."这意味着对于在其范围内的任何内容，Julia 将启动此任务运行，然后继续执行脚本中的下一个内容，而无需等待任务完成.因此，例如，如果没有宏，您将获得:

According to the documentation under ?@async, "@async wraps an expression in a Task." What this means is that for whatever falls within its scope, Julia will start this task running but then proceed to whatever comes next in the script without waiting for the task to complete. Thus, for instance, without the macro you will get:

julia> @time sleep(2)
  2.005766 seconds (13 allocations: 624 bytes)

但是使用宏，您会得到:

But with the macro, you get:

julia> @time @async sleep(2)
  0.000021 seconds (7 allocations: 657 bytes)
Task (waiting) @0x0000000112a65ba0

julia> 

Julia 因此允许脚本继续执行(并且 @time 宏完全执行)而无需等待任务(在本例中，休眠两秒钟)完成.

Julia thus allows the script to proceed (and the @time macro to fully execute) without waiting for the task (in this case, sleeping for two seconds) to complete.

相比之下，@sync 宏将等到 @async、@spawn、 的所有动态封闭使用>@spawnat 和 @parallel 已完成."(根据 ?@sync 下的文档).因此，我们看到:

The @sync macro, by contrast, will "Wait until all dynamically-enclosed uses of @async, @spawn, @spawnat and @parallel are complete." (according to the documentation under ?@sync). Thus, we see:

julia> @time @sync @async sleep(2)
  2.002899 seconds (47 allocations: 2.986 KB)
Task (done) @0x0000000112bd2e00

在这个简单的例子中，将 @async 和 @sync 的单个实例包含在一起是没有意义的.但是，@sync 有用的地方是您将 @async 应用于多个操作，您希望允许所有操作同时开始，而无需等待每个操作完成.

In this simple example then, there is no point to including a single instance of @async and @sync together. But, where @sync can be useful is where you have @async applied to multiple operations that you wish to allow to all start at once without waiting for each to complete.

例如，假设我们有多个工作人员，我们希望让他们每个人同时处理一项任务，然后从这些任务中获取结果.最初(但不正确)的尝试可能是:

For example, suppose we have multiple workers and we'd like to start each of them working on a task simultaneously and then fetch the results from those tasks. An initial (but incorrect) attempt might be:

using Distributed
cell(N) = Vector{Any}(undef, N)

addprocs(2)
@time begin
    a = cell(nworkers())
    for (idx, pid) in enumerate(workers())
        a[idx] = remotecall_fetch(sleep, pid, 2)
    end
end
## 4.011576 seconds (177 allocations: 9.734 KB)

这里的问题是循环等待每个 remotecall_fetch() 操作完成，即每个进程完成其工作(在这种情况下休眠 2 秒)，然后继续开始下一个remotecall_fetch() 操作.就实际情况而言，我们在这里没有获得并行性的好处，因为我们的进程没有同时完成它们的工作(即睡眠).

The problem here is that the loop waits for each remotecall_fetch() operation to finish, i.e. for each process to complete its work (in this case sleeping for 2 seconds) before continuing to start the next remotecall_fetch() operation. In terms of a practical situation, we're not getting the benefits of parallelism here, since our processes aren't doing their work (i.e. sleeping) simultaneously.

不过，我们可以通过组合使用 @async 和 @sync 宏来纠正此问题:

We can correct this, however, by using a combination of the @async and @sync macros:

@time begin
    a = cell(nworkers())
    @sync for (idx, pid) in enumerate(workers())
        @async a[idx] = remotecall_fetch(sleep, pid, 2)
    end
end
## 2.009416 seconds (274 allocations: 25.592 KB)

现在，如果我们将循环的每一步都算作一个单独的操作，我们会看到在 @async 宏之前有两个单独的操作.宏允许其中的每一个启动，并且代码在每个完成之前继续(在这种情况下到循环的下一步).但是，使用 @sync 宏，其范围涵盖整个循环，意味着我们不会允许脚本继续通过该循环，直到 @async 之前的所有操作 已完成.

Now, if we count each step of the loop as a separate operation, we see that there are two separate operations preceded by the @async macro. The macro allows each of these to start up, and the code to continue (in this case to the next step of the loop) before each finishes. But, the use of the @sync macro, whose scope encompasses the whole loop, means that we won't allow the script to proceed past that loop until all of the operations preceded by @async have completed.

通过进一步调整上面的示例以查看在某些修改下它如何变化，可以更清楚地了解这些宏的操作.例如，假设我们只有 @async 而没有 @sync:

It is possible to get an even more clear understanding of the operation of these macros by further tweaking the above example to see how it changes under certain modifications. For instance, suppose we just have the @async without the @sync:

@time begin
    a = cell(nworkers())
    for (idx, pid) in enumerate(workers())
        println("sending work to $pid")
        @async a[idx] = remotecall_fetch(sleep, pid, 2)
    end
end
## 0.001429 seconds (27 allocations: 2.234 KB)

在这里，@async 宏允许我们在每个 remotecall_fetch() 操作完成执行之前继续我们的循环.但是，无论好坏，我们都没有 @sync 宏来防止代码继续通过这个循环，直到所有 remotecall_fetch() 操作完成.

Here, the @async macro allows us to continue in our loop even before each remotecall_fetch() operation finishes executing. But, for better or worse, we have no @sync macro to prevent the code from continuing past this loop until all of the remotecall_fetch() operations finish.

尽管如此，即使我们继续，每个 remotecall_fetch() 操作仍然并行运行.我们可以看到，因为如果我们等待两秒钟，那么包含结果的数组 a 将包含:

Nevertheless, each remotecall_fetch() operation is still running in parallel, even once we go on. We can see that because if we wait for two seconds, then the array a, containing the results, will contain:

sleep(2)
julia> a
2-element Array{Any,1}:
 nothing
 nothing

(nothing"元素是成功获取sleep函数结果的结果，不返回任何值)

(The "nothing" element is the result of a successful fetch of the results of the sleep function, which does not return any values)

我们还可以看到，两个 remotecall_fetch() 操作基本上是同时开始的，因为在它们之前的打印命令也快速连续执行(这些命令的输出未在此处显示).将此与下一个示例进行对比，在该示例中，打印命令以 2 秒的延迟执行:

We can also see that the two remotecall_fetch() operations start at essentially the same time because the print commands that precede them also execute in rapid succession (output from these commands not shown here). Contrast this with the next example where the print commands execute at a 2 second lag from each other:

如果我们将 @async 宏放在整个循环中(而不仅仅是它的内部步骤)，那么我们的脚本将再次立即继续，而无需等待 remotecall_fetch() 操作完成.但是，现在我们只允许脚本作为一个整体继续通过循环.我们不允许循环的每个单独步骤在前一个完成之前开始.因此，与上面的示例不同，在循环后脚本继续执行两秒后，结果数组仍有一个元素为#undef，表示第二个 remotecall_fetch() 操作仍未完成.

If we put the @async macro on the whole loop (instead of just the inner step of it), then again our script will continue immediately without waiting for the remotecall_fetch() operations to finish. Now, however, we only allow for the script to continue past the loop as a whole. We don't allow each individual step of the loop to start before the previous one finished. As such, unlike in the example above, two seconds after the script proceeds after the loop, there is the results array still has one element as #undef indicating that the second remotecall_fetch() operation still has not completed.

@time begin
    a = cell(nworkers())
    @async for (idx, pid) in enumerate(workers())
        println("sending work to $pid")
        a[idx] = remotecall_fetch(sleep, pid, 2)
    end
end
# 0.001279 seconds (328 allocations: 21.354 KB)
# Task (waiting) @0x0000000115ec9120
## This also allows us to continue to

sleep(2)

a
2-element Array{Any,1}:
    nothing
 #undef    

而且，毫不奇怪，如果我们将 @sync 和 @async 放在一起，我们会得到每个 remotecall_fetch() 按顺序(而不是同时)运行，但我们不会在代码中继续运行，直到每个都完成.换句话说，我相信这基本上相当于如果我们没有宏就位，就像 sleep(2) 的行为基本上与 @sync @async sleep(2)

And, not surprisingly, if we put the @sync and @async right next to each other, we get that each remotecall_fetch() runs sequentially (rather than simultaneously) but we don't continue in the code until each has finished. In other words, this would be, I believe, essentially the equivalent of if we had neither macro in place, just like sleep(2) behaves essentially identically to @sync @async sleep(2)

@time begin
    a = cell(nworkers())
    @sync @async for (idx, pid) in enumerate(workers())
        a[idx] = remotecall_fetch(sleep, pid, 2)
    end
end
# 4.019500 seconds (4.20 k allocations: 216.964 KB)
# Task (done) @0x0000000115e52a10

还要注意，在 @async 宏的范围内可能有更复杂的操作.文档 给出了一个示例，其中包含 <代码>@async.

Note also that it is possible to have more complicated operations inside the scope of the @async macro. The documentation gives an example containing an entire loop within the scope of @async.

更新: 回想一下，同步宏的帮助说明它将等到所有动态封闭的 @async、@spawn、@spawnat 和 @parallel 已完成."就什么算作完成"而言，重要的是您如何在 @sync 和 @async 宏的范围内定义任务.考虑下面的示例，它与上面给出的示例之一略有不同:

Update: Recall that the help for the sync macros states that it will "Wait until all dynamically-enclosed uses of @async, @spawn, @spawnat and @parallel are complete." For the purposes of what counts as "complete" it matters how you define the tasks within the scope of the @sync and @async macros. Consider the below example, which is a slight variation on one of the examples given above:

@time begin
    a = cell(nworkers())
    @sync for (idx, pid) in enumerate(workers())
        @async a[idx] = remotecall(sleep, pid, 2)
    end
end
## 0.172479 seconds (93.42 k allocations: 3.900 MB)

julia> a
2-element Array{Any,1}:
 RemoteRef{Channel{Any}}(2,1,3)
 RemoteRef{Channel{Any}}(3,1,4)

前面的示例执行时间大约为 2 秒，表明这两个任务并行运行，并且脚本在继续之前等待每个任务完成其功能的执行.但是，此示例的时间评估要低得多.原因是为了 @sync 的目的，remotecall() 操作一旦将工作发送给工作人员就完成"了.(请注意，此处生成的数组 a 仅包含 RemoteRef 对象类型，这仅表明特定进程正在发生某些事情，理论上可以在将来的某个时间点获取).相比之下，remotecall_fetch() 操作只有在从 worker 收到其任务已完成的消息时才完成".

The earlier example took roughly 2 seconds to execute, indicating that the two tasks were run in parallel and that the script waiting for each to complete execution of their functions before proceeding. This example, however, has a much lower time evaluation. The reason is that for the purposes of @sync the remotecall() operation has "finished" once it has sent the worker the job to do. (Note that the resulting array, a, here, just contains RemoteRef object types, which just indicate that there is something going on with a particular process which could in theory be fetched at some point in the future). By contrast, the remotecall_fetch() operation has only "finished" when it gets the message from the worker that its task is complete.

因此，如果您正在寻找方法来确保在继续执行脚本之前已完成与工作人员的某些操作(例如在这篇文章中讨论的内容:等待任务在 Julia 的远程处理器上完成)它有必要仔细考虑什么算作完整"，以及如何衡量并在脚本中实施它.

Thus, if you are looking for ways to ensure that certain operations with workers have completed before moving on in your script (as for instance is discussed in this post: Waiting for a task to be completed on remote processor in Julia) it is necessary to think carefully about what counts as "complete" and how you will measure and then operationalize that in your script.

这篇关于如何以及何时在 Julia 中使用 @async 和 @sync的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！